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SECTION  I 


INTRODUCTION 


1 . 1  Background 

The  need  for  data  and  information  about  computer  software,  its  development 
process  and  the  software  technology  area  in  general,  has  been  widely  recognized 
by  the  software  community.  Comprehensive  data  and  information  is  needed  to 
provide  a  better  understanding  of  the  factors  contributing  to  the  cost,  pro¬ 
ductivity,  reliability  and  quality  of  software,  to  develop  and  evaluate  better 
and  more  efficient  methods  of  producing  software,  to  predict  costs  of  future 
developments;  to  determine  the  best  design,  development,  and  testing  methodol¬ 
ogies;  and  to  develop  ways  to  effectively  measure  and  predict  various  software 
factors,  such  as  quality,  reliability,  testability,  maintainability,  etc. 

Project  managers  need  data  and  information  from  which  baselines  and  guidelines 
can  be  developed  to  assist  them  in  planning,  measuring  and  tracking  of  software 
development  projects.  Furthermore,  with  the  rapid  expansion  of  software  engi¬ 
neering  technology  and  the  proliferation  of  tools  and  techniques,  it  has  been 
extremely  difficult  for  an  individual  or  organization  to  remain  cognizant  of 
the  current  status  of  the  software  engineering  field.  This  situation  has 

resulted  in  duplication  of  efforts  in  software  research  and  has  seriously  ham¬ 

pered  the  transfer  of  technology  from  the  software  research  environment  to  the 
user  sector  of  the  software  community. 

The  Air  Force  recogni zed  the  need  for  an  information  analysis  center  to 
serve  the  government,  industrial,  and  university  community  as  a  focal  point  for 
software  development  and  experience  data.  The  Rome  Air  Development  Center 
( RADC )  contracted  with  1 1 T  Research  Institute  (IITRI)  to  design  such  a  center 
that  would  acquire,  analyze,  synthesize,  and  dissemi nate  i nformation  on  software 
engineering  technology,  (reference  1).  Subsequently,  in  August  of  1978,  RADC 
contracted  with  IITRI  to  develop  such  a  center,  named.  The  Data  and  Analysis 
S  Center  for  Software  (DACS).  It  is  the  purpose  of  this  report  to  record  the 

|  0  activities,  accomplishments,  and  status  of  the  development  of  the  DACS  during 

I  ■  its  first  18-month  period  (August  1978-February  1980). 

*  The  DoD  and  other  Federal  Agencies  have  found  that  the  establishment  of 

„  special  purpose  information  analysis  centers  and  technology  transfer  programs 

has  been  effective  in  overcoming  problems  in  technology  implementation  and 
diffusion  of  mission-oriented  developments.  NASA,  for  example,  has  been  able 
to  demonstrate  a  benefit- to-cost  ratio  in  excess  of  ten  to  one  for  its  Tech¬ 
nology  Utilization  Program.  Similarly,  the  DoD  has  sucessfully  operated  Infor¬ 
mation  Analysis  Centers  for  Metals,  Ceramics,  Hardware  Reliability,  and  Machin- 
ability  data,  (reference  18). 

1 . 2  Objectives  of  the  Center 

When  fully  implemented  and  operational,  the  DACS  will  provide  a  centralized 
source  for  current,  and  readily-usable  data  and  information  concerning  software 
technology.  The  objectives  of  the  software  information  resource  are: 
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•  Encourage  the  diffusion  of  technology  to  DoD,  Civil  Agencies,  Govern¬ 
ment  contractors,  etc. 

•  Bring  about  higher  levels  of  utilization  of  project  results  in  a  cost 
effective  manner. 

•  Increase  the  productivity  and  quality  of  computer  software  by  improv¬ 
ing  the  transfer  of  software  engineering  technology. 

•  Assist  in  diffusing  new  technology  throughout  the  U.S.  industrial 
base,  thereby  expanding  its  capability  and  competitive  posture. 

•  Provide  scientific  and  technical  information  analysis  services  to 
DoD,  Civil  Agencies,  government  contractors,  and  the  private  sector 
in  areas  relating  to  software  technology  needs,  developments,  and 
trends. 

•  Minimize  duplication  of  effort  thereby  reducing  costs. 

1 . 3  Report  Contents 

This  report  provides  a  summary  of  the  development  of  the  DACS  with  these 
objectives  in  mind  and  contains  eight  sections  and  three  appendices.  The  devel¬ 
opment  activities  of  the  DACS  were  oriented  toward  building  an  information  base 
on  software  engineering  technology  and  transferring  information  about  the  tech¬ 
nology  in  a  form  that  can  readily  be  understood,  evaluated,  and  used. 


The  following  is  a  short  description  of  the  various  areas  covered  in  the 
specific  sections. 


Section  I 
Section  II 
Section  III 

Section  IV 

Section  V 


Section  VI 


Section  VII 

Section  VIII 
Appendix  A 
Appendix  B 

Appendix  C 


Background  and  objectives  of  the  Center 
Summary  of  Technical  progress  and  activities 
Descriptions  of  the  DACS  Database  and  data  and  information 
synthesis  activities 

Composites  of  the  newsletters,  information  publications, 
and  technical  symposia  activities 

Descriptions  of  the  software  engineering  technical  infor¬ 
mation  base  including  the  DACS  library,  the  bibliographic 
database,  the  DACS  Thesaurus,  and  the  DACS  Glossary 
Discussions  of  the  various  DACS  products  and  services 
including  the  data  subsets,  a  data  compendium,  technical 
reports,  and  technical  inquiries 

The  results  of  a  DACS  user-survey  and  analysis  of  inquir¬ 
ies  processed 

Conclusions  and  recommendations 
A  partial  list  of  DACS  Users 

Statistics  on  the  utilization  of  the  keywords  in  the 
DACS  Thesaurus 

A  concept  paper  discussing  the  role  of  the  DACS  in  trans- 
fering  software  engineering  technology 
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SECTION  II 


* 


TASK  1  ESTABLISH  AND  OPERATE  CENTER 


2.1  Summary  of  Technical  Progress  and  Activities 

2.1.1  Approach 

The  Data  and  Analysis  Center  for  Software  is  being  developed  on  a  build 
approach  basis.  That  is,  an  information  base  is  being  established  and  used  to 
provide  incremental  additions  of  products  and  services  to  demonstrate  the 
center's  immediate  and  growing  effectiveness.  This  build  approach  is  highlighted 
by  dividing  the  descriptions  of  the  progress  to  date  and  the  progress  planned 
into  three  phases. 

2.1.2  Summary  by  Phase 

Table  2.1-1  lists  the  major  milestones  for  each  phase  of  the  development 
of  the  DACS.  Phase  I  covers  the  75s-month  period  from  the  initiation  of  the  con¬ 
tract  (15  August  1978)  to  1  April  1979.  This  phase  included  establishing  com¬ 
munication  channels  with  the  software  technology  community,  generating  the  pro¬ 
cedures  (both  manual  and  automated)  for  developing  the  technology  information 
base,  initiating  this  development,  producing  a  state-of-the-art  report  on  quan¬ 
titative  software  models,  and  answering  91  informational,  technical,  and  docu¬ 
ment  inquiries. 

Phase  II  (1  April  1979-15  February  1980)  included  a  continuation  of  many 
of  the  activities  performed  under  Phase  I,  as  illustrated  in  Table  2.1-1.  In 
addition  to  these  continued  activities,  a  DACS  Glossary,  a  set  of  data  collec¬ 
tion  forms,  a  state-of-the-art  report  on  software  maintenance,  and  a  data  com¬ 
pendium  were  produced.  The  total  number  of  inquiries  received  and  processed 
during  this  10-month  period  was  2,162. 

By  the  end  of  Phase  I  and  Phase  II  (which  is  this  reporting  period)  we 
have  accomplished  the  following  objectives: 

t  Provided  an  awareness  of  the  DACS  to  the  software  technology 
communi ty . 

•  Produced  products  and  provided  services  that  were  of  immediate  use 
to  the  community. 

•  Established  a  technology  information  base  to  be  used  as  a  framework 
for  analysis,  consolidation,  and  synthesis. 

Phase  III  of  this  development  of  the  DACS  (15  February  1980-15  August  1981) 
will  concentrate  on  providing  more  quality  and  indepth  products  and  services. 

An  active  data  acquisition  program  will  be  initiated,  and  a  cost  recovery  pro¬ 
gram  will  be  developed.  The  technology  information  base  will  be  expanded  and  more 
engineering/data  analysis  will  be  performed  to  provide  synthesized  information 
to  researchers,  managers,  and  software  developers. 
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TABLE  2.1-1  CENTER  ACTIVITIES  BY  PHASE 


’base  ;  1  Phase  II 

;  T5  August  1 3 78- ’  April  1979)  (1  April  1979-15  FeDruary  1980! 

?'i- Month  p?noa  lO^-Month  Perioa 

Phase  III 

(15  February  1980-15  August  1981) 
18-Month  Period 

Established  physical  facilities. 
Establisned  mecnanisms  for  pro¬ 
cessing  requests. 

Initiated  document  processing. 

Refined  procedures. 

Distributed  and  analyzed  the  DACS 

User  Survey. 

Continue  refining  procedures. 
Develop  ano  implement  an  active 
data  acquisition  program. 

Develop  a  cost  recovery  program. 

Compiled  newsletter  mailing  list, 
i.  21  *3  names/ organ!  zationsi 

Updated  newsletter  mailing  list. 

(2619  names/orqanizations) 

Maintain  Newsletter  mailing  list. 

Generated  'hesaurus. 

Produced  the  first  DACS  Glossary. 

°roouce  the  second  OACS  Glossary. 

Preoared  ana  als ’.ributea  six 

TKjntnly  News  letters. 

Prepared  and  distributed  three 
quarterly  newsletters  and  aignt 

Sul  leti ns. 

Preoare  and  distribute  si*  quar- 
ter'y  Newsletters  ana  twelve 

Bui  leti  ns. 

Produced  tne  Data  3arameters 

Reoort. 

Produced  and  distributed  a  set  of 
productivity  data  collection 
forms . 

Establish  project  status 
capability. 

Organized  tne  IEEE  Terminology 

Task  Group. 

Coordinated  activities  of  the  IEEE 
Terminology  Task  Group. 

— 

Coordinate  production  of  the  IEEE 
terminology  document. 

1 

Prepared  and  Dresented  intro¬ 
ductory  oapers  on  the  DACS 

. 

Prepared  and  presented  papers  on 
the  OACS,  quantitative  software 
models,  database  evaluation,  and 
software  reliability. 

Continue  oresenting  oapers,  etc. 

Initiated  the  acquisition  of  the 

NASA/ S EL  database. 

i 

! 

1 

Implemented  the  MASA/SEL  database  on 
the  RAOC  Computer.  Produced  a  data 
compendi un  based  on  this  database. 

Distributed  copies  of  the  RADC  Pro¬ 
ductivity  Oatabase. 

Implement,  analyze  and  distribute 
databases  as  received. 

Produce  the  second  data 
compendium. 

Designed  and  'implemented  the  data- 
#1’es  "or  the  Glossary,  Thes- 
j  aurus,  snd  bibliographic  data¬ 
base. 

1  Generated  comouter  queries. 

Refined  and  distributed  the  DACS 
Glossary  and  the  Thesaurus. 

Produced  custom  bibliographic 
searches. 

Continue  maintaining  technology 
information  base. 

0 repare  two  comprehensive  software 
engineering  bibliographies. 

Produced  snd  distributed  the  users 
guide,  ’'Bibliographic  Services, 

Custom  Searches"  (SIB-I). 

Perform  engineering  research,  dis¬ 
till  and  synthesize  biblio¬ 
graphic  and  database  in  formation 

Produced  SOA  Peoort  "Duanti  tative 
Software  Models"  ,'SRR-l). 

Produced  SOA  Report  "Software  Main¬ 
tenance  Technology." 

Produce  one  SOA  Report  and  two 
technical  monographs. 

Processed  95  technical  and  docu¬ 
ment  inquiries. 

1  ... 

Processed  !as  of  1  February)  281 
technical  inquiries,  '212  docu¬ 
ment  inquiries,  and  669  News- 
letter  Inquiries. 

_ 

Continue  processing  tecnnieal  and 
document  inquiries. 
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SECTION  III 


DATABASE  AND  TECHNOLOGY  ANALYSIS 


3.1  Introduction 


This  section  of  the  report  contains  descriptions  of  the  DACS  software 
experience  database  and  the  results  of  analyses  performed  in  the  areas  of  data 
collection,  database  evaluation,  quantitative  software  models,  and  software 
maintenance. 

3-2  Task  2  Computer  Database 

3.2.1  Introduction 

The  DACS  Computer  database  presently  consists  of  four  sets  of  data  distin¬ 
guishable  by  data  source,  data  collection  and  acquisition  methodology,  and  data 
contents.  These  four  sets  of  data  are  described  in  subsequent  sections  of  this 
report  and  are  termed: 

§  Baseline  Software  Database 

•  RADC  Productivity  Database 

•  Software  Reliability  Database 

•  NASA  SEL  Database 

These  databases  have  been  implemented  on  the  RADC  HIS  6180  computer  system 
using  the  Management  Data  Query  System  (MDQS)  for  data  retrieval  and  analysis 
and  for  producing  subsets  of  the  data. 

3.2.2  Baseline  Software  Database 

This  DACS  database  consists  of  six  distinct  datasets  which  contain  data 
describing  software  problem  reports  acquired  by  RADC  from  six  large  software 
development  efforts.  These  projects  and  the  data  are  described  in  references 
2  through  7. 

The  system  application  areas  encompass  command  and  control,  real-time  con¬ 
trol  for  land-based  radar,  onboard  guidance  and  navigation,  and  database  manage¬ 
ment.  The  majority  of  the  datasets  contain  the  date  the  software  problem  was 
opened  and  closed,  the  module  that  manifested  the  problem,  the  module  that  was 
changed  to  correct  the  problem,  the  problem  category  and  priority,  and  the  cor¬ 
rection  type. 

Three  of  the  datasets  contain  module  descriptive  information  which  desig¬ 
nates  the  number  of  source  instructions,  the  programming  language,  the  type  of 
construction,  and  a  functional  area  designation.  One  dataset  records  information 
on  each  test  run. 

This  database  presently  consists  of  26,594  Software  Problem  Report  records, 
2719  Run  Analysis  Report  records,  and  2591  Module  Description  records.  Refer¬ 
ences  8  and  9  describe  the  implementation  of  these  datasets  on  the  RADC  computer 
system. 
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This  database  was  compiled  by  Richard  Nelson  of  RADC  and  contains  suninary 
information  from  over  400  software  projects,  some  dating  back  to  the  early  six¬ 
ties,  mostly  on  DoD  applications.  The  Database  incorporates  productivity  data, 
error  data,  project  duration,  total  effort,  language  data,  and  information  on 
the  usage  of  various  software  implementation  technologies.  At  present,  93%  of 
the  projects  contain  productivity  data,  7%  contain  error  data,  91%  contain 
language  usage  data  and  about  50%  contain  information  on  implementation  tech¬ 
nologies.  In  total,  this  database  covers  approximately  21  million  lines  of 
code.  The  data  has  been  collected  from  a  large  variety  of  sources,  both  public 
and  private. 

A  preliminary  analysis  and  a  description  of  this  database  is  contained  in 
reference  10. 

3.2.4  Software  Reliability  Database 

This  database  was  provided  to  the  DACS  by  John  D.  Musa,  Bell  Telephone 
Laboratories,  Whippany,  NJ,  wi th  the  hope  that  it  will  be  of  general  benefit  for 
software  reliability  researchers  and  that  it  will  permit  independent  verifica¬ 
tion  of  his  conclusions  on  the  execution  time  theory  of  software  reliability, 
(reference  11). 

The  primary  objective  for  collecting  this  data  on  16  software  systems  was 
to  assist  software  managers  in  monitoring  status  and  predicting  schedules. 
Careful  controls  were  employed  to  ensure  that  the  data  would  be  of  high  quality. 

Failure  interval  data  (failure  number,  failure  interval,  day  of  failure) 
is  provided  for  all  16  systems  with  a  total  sample  size  of  2831.  Failure  cor¬ 
rection  dates  and  resource  expenditure  information  is  available  for  four  of  the 
systems. 

The  size  of  the  software  systems  vary  from  approximately  6,000  to  hundreds 
of  thousands  delivered  object  instructions.  The  nature  of  the  systems  are  real 
time  command  and  control,  real  time  commercial,  operating  systems,  a  time  shar¬ 
ing  system,  and  a  word  processing  system.  Operational  failure  data  is  available 
from  11  of  the  systems,  system  test  failure  data  from  eight  of  the  systems,  and 
subsystem  test  failure  data  from  one  of  the  systems.  Four  of  these  datasets 
contain  failure  and  resource  expenditure  data  for  both  the  system  test  and 
operational  phases  of  the  software  life  cycle. 

3.2.5  NASA  SEL  Database 

The  Software  Engineering  Laboratory  (SEL),  at  NASA  Goddard  Space  Flight 
Center,  was  organized  in  August  1976  to  monitor  existing  software  methodologies 
and  to  develop  and  measure  the  effectiveness  of  alternative  methodologies, 
(reference  12).  To  accomplish  these  objectives,  they  have  been  collecting  data 
during  the  development  of  their  software  products  and  have  submitted  their  data 
to  the  DACS  for  subsequent  analysis  and  distribution.  This  database  will  con¬ 
tinue  to  be  updated  as  more  data  is  collected. 
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As  of  1  February  1980  the  NASA  SEL  Database  at  the  DACS  contained  infor¬ 
mation  on  29  software  development  projects.  These  projects  include  orbit  and 
attitude  determination  systems,  mission  support  packages,  attitude  ground  sup¬ 
port  systems,  a  human  resources  scheduling  system,  a  FORTRAN  source  analyzer, 
and  other  software  support  systems.  The  computer  programs  were  written  in 
FORTRAN  or  a  combination  of  FORTRAN  and  assembly  language  for  a  PDP  11/70  or  a 
System  360. 

The  majority  of  the  data  is  from  run  analysis  reports  (over  10,000  records). 
The  remainder  is  information  from  component  status  reports  (3760  records),  pro¬ 
ject  comment  information  (2994  records),  change  reports  (1172)  records),  resource 
summary  reports  (262  records),  and  component  summary  reports  (208  records). 

3. 3  Task  3  Data  Analysis  Program 

3.3.1  Introduction 

The  major  activity  performed  under  this  task  definition  was  the  develop¬ 
ment  of  a  database  evaluation  methodology.  This  methodology  was  developed  and 
used  to  determine  the  relevance  and  limitations  of  the  Baseline  Software  Data¬ 
base  to  perform  research  in  software  error  analysis  and  reliability  modeling. 

3.3.2  Database  Evaluation 

Quantitative  metrics  to  measure  and  evaluate  the  quality  of  existing  and 
future  data  for  modeling  and  analysis  efforts  were  developed.  These  metrics 
will  facilitate  data  collection  efforts  and  modeling  activities. 

The  metrics  developed  are  named  integration  and  coverage.  Integration  (I) 
is  a  numeric  measure  of  the  similarity  between  the  datasets  in  a  database.  A 
high  degree  of  similarity  (X=  1)  facilitates  comparisons  of  results  of  a  given 
model  among  different  datasets  in  a  database.  Coverage  is  a  graphical  metric 
which  measures  what  model  can  be  driven  by  the  data  contained  in  a  database. 

In  this  technique,  the  cells  of  a  two-dimensional  array  (whose  axes  are  (x) 
model  and  (y)  data  parameter)  are  filled  with  the  name  of  the  dataset  and  data 
element  that  can  supply  the  data  parameter  for  the  model.  Matches  between  data 
parameters  and  data  elements  can  then  be  determined. 

These  metrics  were  used  to  determine  the  adequacy  of  the  data  in  the  Base¬ 
line  Software  Database  to  do  comparisons  across  a  wide  variety  of  projects  and 
to  determine  if  the  database  contains  data  elements  as  required  by  the  various 
software  reliability  models  and  other  analytical  techniques.  A  complete  dis¬ 
cussion  of  this  methodology  and  the  results  of  the  evaluation  of  the  Baseline 
Software  Database  is  included  in  reference  13. 

The  evaluation  determined  that  the  integration  factor  for  the  Baseline 
Software  Database  was  1^  =  0.28,  a  low  value  that  indicates  that  this  database 
is  not  well  suited  to  cross  comparisons  among  datasets.  Database  coverage  is 
high  for  models  that  utilize  software  problem  and  software  modification  data, 
but  there  are  a  variety  of  error  analysis  and  reliability  models  that  cannot 
fulfill  their  data  requirements  from  the  available  data  elements. 
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3.4  Task  4  Data  Requirements 


3.4.1  Introduction 

This  section  of  the  report  summarizes  the  results  of  a  study  to  determine 
the  kinds  of  data  required  to  utilize  software  models  and  assist  modeling  re¬ 
search  efforts.  Also  included  is  a  description  of  data  collection  forms  which 
were  developed  to  provide  a  basis  for  collecting  software  productivity 
i nformation. 

3.4.2  Data  Parameters  Report 

A  study  was  performed  to  determine  the  data  parameters  needed  to  support 
research  activities,  model  developments,  and  analysis  efforts  in  the  areas  of 
software  research  (software  reliability,  software  error  analysis,  life-cycle 
cost  analysis,  software  productivity,  and  complexity).  A  report  was  produced 
that  contains  descriptions  of  software  models  and  the  kinds  of  data  required  for 
each  model.  The  model  descriptions  include: 

•  Category  of  the  model 

•  Sources  referenced 

f  Brief  statement  of  the  purpose  and  characteristics  of  the  model 

•  An  enumeration  of  the  significant  metrics 

•  Formulas  or  other  outputs  of  the  model 

The  model  descriptions  for  life-cycle  cost/productivity,  reliability/error 
analysis,  and  complexity  are  contained  in  three  separate  sections.  Each  section 
ends  with  a  matrix  which  cross  references  the  models  and  their  parameters. 

3.4.3  Productivity  Data  Collection  Forms 


Two  data  collection  forms  were  designed  for  software  life-cycle  productiv¬ 
ity-related  data.  The  first  form,  the  Project  Summary  Form,  describes  a  pro¬ 
ject's  software  development  environment.  The  second  form  is  the  Component 
Summary  and  describes  the  software  components  of  a  project,  resources  expended 
during  its  development,  and  any  noted  discrepancies.  Component  data  may  be 
collected  at  several  levels  such  as  System,  Functional  Group,  and  Module.  In 
general,  the  forms  have  been  designed  to  identify  key  data  collection  areas  yet 
be  flexible  enough  to  accommodate  additional  project-specific  data. 

The  Project  Summary  Form  is  used  to  collect  data  concerning  (1)  the  gen¬ 
eral  nature  of  the  project,  (2)  the  project  priorities  and  constraints,  and  (3) 
the  software  development  technologies  used  in  the  project.  The  data  elements 
of  the  project  name,  project  type,  project  description,  number  of  systems, 
number  of  functional  groups,  number  of  modules  and  standards  record  the  general 
nature  of  the  project.  Priorities  and  constraints  may  be  chosen  from  several 
common  ones  listed  on  the  form,  or  project-specific  entries  may  be  noted.  A 
list  of  software  development  technologies  is  also  provided.  Additional  entries 
may  be  added. 
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The  Component  Summary  Form  is  used  to  collect  data  concerning  (1)  the 
software  engineering  characteristics  of  the  component,  (2)  the  resources  ex¬ 
pended  throughout  the  life  cycle  of  the  component  and  (3)  the  discrepancies 
detected  in  the  component's  life  cycle.  Complexity,  type,  description,  size, 
programming  language,  and  mode  of  construction  describe  the  component  in  soft¬ 
ware  engineering  terms.  The  resources  expended  on  the  component  are  recorded 
by  life-cycle  phase  and  time  in  terms  of  person-hours  and  computer- hours.  The 
number  of  discrepancies  found  in  the  component  are  recorded  by  life-cycle  phase 

The  purpose  of  these  forms  is  to  collect  project  and  component  level  data 
that  is  common  to  most  software  development  projects.  If  these  forms  are  to  be 
used  as  a  tool  for  collecting  data  for  a  specific  quantitative  software  engi¬ 
neering  model  or  method,  additional  project-specific  forms  should  be  developed. 
To  develop  these  special  forms,  DACS  Software  Engineering  Research  Review  - 
Quantitative  Software  Models,  SRR-1,  should  be  used  to  determine  the  data  para¬ 
meters  to  be  collected  for  the  specific  application. 

These  forms  and  their  instructions  are  being  distributed  by  the  DACS  to 
invite  feedback  from  the  software  research  and  development  community. 

3.5  Task  5  Input  Processing  -  Software  Data 

This  task  consisted  of  processing  the  Software  Reliability  and  NASA/SEL 
Databases.  These  databases  were  reviewed  and  evaluated,  loaded  onto  the  RADC 
Honeywell  6180  Computer  System,  and  defined  using  the  data  definition  languages 
of  MDQS. 
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SECTION  IV 

TASK  6  CURRENT  AWARENESS  PROGRAM 


4.1  Introduction 


The  purpose  of  this  task  is  to  inform  users  of  the  services  of  the  Center 
and  to  provide  the  software  community  with  up-to-date  information  on  software 
technology.  Six  monthly  newsletters,  three  quarterly  newsletters,  and  eight 
bulletins  were  produced  and  distributed  during  the  year  and  a  half  period. 
Descriptive  information  on  the  DACS  was  prepared  and  distributed  in  response 
to  requests  for  information  on  the  Center.  Sixteen  presentations  were  made  at 
conferences,  workshops,  and  technical  meetings  to  discuss  the  DACS  and  summarize 
technical  results.  These  activities  are  discussed  below. 

4.2  Newsletters  and  Bulletins 


The  contents  of  the  DACS  monthly  newsletters,  quarterly  newsletters,  and 
bulletins  are  summarized  in  Tables  4.2-1,  4.2-II,  and  4.2-III,  respectfully. 

The  newsletters  and  the  bulletins  are  both  informative  and  promotional.  They 
have  contained  information  on  software  technology  activities,  report  reviews, 
synopses  of  symposia,  referral  information,  and  descriptions  of  DACS  products 
and  services.  As  of  1  February  1980,  there  were  2619  names  on  the  newsletter 
mailing  list.  The  bulletin  is  distributed  to  approximately  50  individuals. 

4. 3  Information  Publications 

Literature  on  the  DACS  products  and  services  has  been  prepared  and  distri¬ 
buted  to  users  and  potential  users  of  the  center.  A  general  information  packet 
is  sent  in  response  to  an  initial  inquiry  on  the  DACS.  This  packet  contains  a 
copy  of  the  paper  entitled  "An  Introduction  to  the  Data  and  Analysis  Center  for 
Software,"  individual  flyers  which  contain  order  forms  and  describe  the  various 
products,  and  the  guide  entitled  "  Bibliographic  Services,  Custom  Searches." 

This  guide  (reference  BIB-1)  was  designed  to  acquaint  the  users  of  the  DACS 
with  the  procedures  required  to  obtain  a  custom  search  of  the  DACS  bibliographic 
data  files.  As  of  1  February  1980,  817  copies  of  BIB-1  had  been  distributed. 

4.4  Technical  Symposia  Activities 

Sixteen  technical  presentations  were  made  at  software  related  technical 
symposia,  and  are  listed  in  Table  4.4-1.  This  list  includes  seven  papers  that 
were  published  in  the  workshop  or  conference  proceedings.  The  major  subject 
areas  included  in  these  papers  are: 

•  An  introduction  and  status  of  the  DACS 

•  A  summary  of  the  DACS  report  entitled  "Quantitative  Software  Models" 

•  The  state-of-the-art  in  software  reliability 

•  A  review  of  the  DACS  database  evaluation  methodology 

•  Software  engineering  technology  transfer 
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1979  1978 


TABLE  4.2-1  MONTHLY  DACS  NEWSLETTERS 


Date 

Information  Highlights 

October 

Introductory  information  on  the  DACS 

November 

Synopses  of  three  software  technology 
workshops 

December 

A  discussion  on  six  of  the  DACS  datasets, 
and  reviews  of  two  software  reports  related 
to  software  data  collection  and  software 
rel iabi li ty 

January 

Definitions  of  software  reliability  terms, 
a  discussion  on  software  reliability  theory, 
and  a  book  review  on  software  metrics 

February 

Summaries  of  two  software  related  facilities, 
descriptions  of  software  engineering  com¬ 
mittees,  and  synopses  of  two  symposia 

March 

Descriptions  of  the  DACS  bibliographic 
services  and  SRR-1,  and  a  profile  of  DACS 
inqui ries 

TABLE  4. 2-1 1 


QUARTERLY  DACS  NEWSLETTERS 


Date 

Information  Highlights 

June 

Three  document  reviews  on  computer  program 
and  database  conversion,  summary  statistics 
from  the  RADC  Productivity  Database,  and 
workshop  announcements 

1979 

September 

Information  on  obtaining  the  RADC  Produc¬ 
tivity  Database  and  the  DACS  data  collection 
forms,  on  surveys  being  conducted  on  soft¬ 
ware  tools,  on  the  availability  of  several 
software  guidebooks,  a  summary  of  the  DACS 
bibliographic  search  services,  and  the  DACS 
User  Survey. 

December 

A  discussion  on  the  history  and  status  of 
the  DACS,  a  listing  of  products,  an  annouce- 
ment  of  the  DACS  glossary  and  the  software 
reliability  database,  summaries  of  two  con¬ 
ferences  and  one  report,  and  a  listing  of 
centers  and  services  that  catalog  and/or 
distribute  computer  programs 
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TABLE  4. 2-1 1 1  DACS  BULLETINS 


Date 

Information  Highlights 

Apri  1 

The  results  of  a  custom  bibliographic  search 
on  software  development  tools  and  techniques 

May 

A  discussion  of  the  DACS  software  engineering 
database  evaluation  methodology 

or 

r^ 

cn 

July 

The  DACS  Productivity  Forms  with  instruc- 
ti  ons 

August 

Announcement  of  the  NBS  and  AIAA  software 
tools  surveys 

October 

A  synopsis  of  the  Second  Minnowbrook  Work¬ 
shop  on  Software  Performance  Evaluation 

November 

History  and  status  of  the  DACS 

§ 

January 

A  summary  of  GAO  Reports  related  to  soft¬ 
ware  technology 

o> 

February 

A  review  of  two  reports  on  software  quality 
metri cs 

14 
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SECTION  V 


TASK  7  TECHNOLOGY  INFORMATION  BASE 


5.1  Introduction 


One  of  the  functions  of  the  DACS  is  the  establishment  of  a  software  tech¬ 
nology  scientific  and  technical  information  base.  During  this  reporting  period, 
we  have  produced  a  software  engineering  thesaurus,  a  glossary  of  software  engi¬ 
neering  terms  compiled  from  the  literature,  and  a  software  engineering  document 
library  including  a  computer  database  containing  files  of  document  descriptions 
and  bibliographic  citations. 

The  status  of  this  information  base  is  discussed  in  this  section  of  the 
report. 

5 . 2  The  Software  Engineering  Document  Library 

The  DACS  Library  has  been  established  to  provide  a  readily  accessible 
source  of  comprehensive  information  on  the  state-of-the-art  in  software  engi¬ 
neering  as  well  as  a  means  of  channeling  that  information  to  those  people  in 
the  software  engineering  community  who  can  make  use  of  it  in  their  day-to-day 
activities  of  developing,  maintaining,  and  managing  software.  The  biblio¬ 
graphic  collection  is  composed  of  texts,  technical  reports,  theses,  journal 
articles,  proceedings  and  other  documents  relating  to  software  engineering, 
reliability,  cost  and  quality  factors ,  maintainability,  and  other  topics  deemed 
appropriate. 

As  of  1  February  1980,  the  DACS  has  approximately  2200  documents  relating 
to  software  engineering;  1346  of  these  document  have  been  indexed,  their  biblio¬ 
graphic  citations  coded,  keypunched  and  placed  online.  Table  5.2-1  lists  the 
types  of  documents  in  the  online  portion  of  the  collection,  the  number  of  docu¬ 
ments  of  each  type  and  the  percent  for  each  type.  Note  that  85%  of  the  biblio¬ 
graphic  collection  is  made  up  of  journal  articles  and  papers.  Most  of  the 
papers  have  been  presented  at  various  conferences  and  symposia  and  have  been 
published  in  the  proceedings  cf  those  conferences  and  symposia. 

A  full  listing  of  the  various  journals  and  proceedings  from  which  the  DACS 
has  obtained  articles  and  papers  is  contained  in  table  5.2-II. 

The  DACS  Software  Engineering  Bibliographic  collection  is  made  available 
to  members  of  the  software  engineering  community  chiefly  in  the  form  of  biblio¬ 
graphies.  The  DACS  functions  as  a  reference  library  in  that  custom  searches 
are  made  upon  receipt  of  a  request  from  a  user.  The  request  may  be  made  during 
a  visit  to  the  center,  by  a  telephone  call,  or  by  a  written  request.  When  the 
search  has  been  completed,  the  resulting  bibliography  is  given  or  sent  to  the 
user  who  made  the  request. 
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Bibliography 

Journal  Article 

Technical  Report 

Text 

Paper 

Standard 

Regulation 

Specification 

Monograph 


TABLE  5.2-1  DOCUMENT  TYPE 


Number  of  Documents 

Percent  of  Collection 

0006 

.4 

0253 

18.8 

0156 

11.6 

0008 

.6 

0898 

66.7 

0008 

.6 

0003 

.2 

0003 

.2 

0011 

.8 

17 

T, 


TABLE  5. 2- II 


BREAKDOWN  STATISTICS  FOR  JOURNAL  ARTICLES  AND  PAPERS 
AS  OF  02/07/80 

NAME  OF  JOURNAL  OR  CONFERENCE  NO.  PAPERS 

OR  ARTICLES 

ACM  78, PROCEEDINGS  OF  THE  1978  ANNUAL  CONFERENCE  0036 

ACM  COMPUTING  SURVEYS  0005 

AFIPS  NAT'L  COMPUTER  CONFERENCE  0018 

PROCEEDINGS  OF  THE  SECOND  ARMY  SOFTWARE  SYMPOSIUM  0028 

BELL  SYSTEM  TECHNICAL  JOURNAL  0001 

A I AA/NAS A/ IEEE /ACM  COMPUTERS  IN  AEROSPACE  CONFERENCE  0058 

COMPUTERS  AND  PEOPLE  0002 

COMPCON  ' 78  0001 

1976  CONFERENCE  ON  INFORMATION  SCIENCES  &  SYSTEMS  0001 

PROCEEDINGS .COMPSAC  77  0050 

PROCEEDINGS  .COMPSAC  78  0100 

COMPUTER  0049 

COMPUTERWORLD  0018 

COMPUTER  BUSINESS  NEWS  0003 

FIRST  CANADIAN  SRE  RELIABILITY  SYMPOSIUM,  1974  0001 

1975  CANADIAN  SRE  RELIABILITY  SYMPOSIUM  0001 

ANN.  COMPUTER  RELATED  INFORMATION  SYSTEMS  SYMP.,1977  0006 

ANN.  COMPUTER  RELATED  INFORMATION  SYSTEMS  SYMP.,1978  0005 

ANN.  COMPUTER  RELATED  INFORMATION  SYSTEMS  SYMP.,1979  0008 

COMPUTER  SYSTEMS  NEWS  0001 

1973  IEEE  SYMPOSIUM  ON  COMPUTER  SOFTWARE  RELIABILITY  0025 

3RD  DIGITAL  AVIONICS  CONFERENCE  0001 

DATABASE  00C2 

DATAMATION  0012 

EASCON  ’74  0002 

ELECTRONICS  0006 

ELECTRONIC  DESIGN  0009 

1972  FALL  JOINT  COMPUTER  CONFERENCE  0001 

HARVARD  BUSINESS  REVIEW  0001 

INFOR  (CANAD.JRNL  OF  OPERATIONAL  RES.&  INFOR.PROC.)  0001 

INFORMATION  PROCESSING  74  0003 

4TH  INT'L  CONFERENCE  ON  SOFTWARE  ENGINEERING  0016 

2ND  SOFTWARE  LIFE  CYCLE  MANAGEMENT  CONFERENCE  0001 

MICRO  &  MINI  SYSTEMS,  SYMP.  ON  TRENDS  &  APPL.,1976  0012 

MILITARY  ELECTRONICS/COUNTERMEASURES  0002 

NAECON  '75  RECORD  (NATL  AVIONICS  &  ELECTRICAL  CONF)  0005 

NATIONAL  COMPUTER  CONFERENCE,  1974  0002 

NATIONAL  COMPUTER  CONFERENCE,  1975  0001 


TABLE  5. 2-1 1  con't 


BREAKDOWN  STATISTICS  FOR  JOURNAL  ARTICLES  AND  PAPERS 
AS  OF  02/07/80 


NAME  OF  JOURNAL  OR  CONFERENCE  NO.  PAPERS 

OR  ARTICLES 

2ND  NATL  RELIABILITY  CONF., 1979  (BIRMINGHAM,  ENGLAND)  0003 

ACM/NBS  18TH  ANN.  TECHNICAL  SYMP. .INFORMATION  SYSTEMS  0013 

PROCEEDINGS  OF  THE  IEEE,  1975  0001 

PRACTICAL  STRATEGIES  FOR  DEVELOP'G  LG  SOFTWARE  SYSTEMS  0008 

PROCEEDINGS,  1976  RAC  RELIABILITY  WORKSHOP  0001 

RESEARCH  DIRECTIONS  IN  SOFTWARE  TECHNOLOGY  0055 

1975  ANNUAL  RELIABILITY  &  MAINTAINABILITY  SYMPOSIUM  0005 

1979  ANNUAL  RELIABILITY  AND  MAINTAINABILITY  SYMPOSIUM  0004 

PROCEEDINGS, 1975  I NT  *  L  CONFERENCE  ON  RELIABLE  SOFTWARE  0059 

MR I  SYMPOSIUM  ON  COMPUTER  SOFTWARE  ENGINEERING  1976  0038 

2ND  INT'L  CONFERENCE  ON  SOFTWARE  ENGINEERING  0084 

3RD  INT'L  CONFERENCE  ON  SOFTWARE  ENGINEERING  0047 

2ND  INT'L  CONF.  ON  SOFTWARE  ENGINEERING,  ADDENDUM  0018 

1ST  NATL.  CONFERENCE  ON  SOFTWARE  ENGINEERING  (1975)  0014 

SIGNAL  -  JOURNAL  OF  AFCEA  0001 

1977  ANN.  SOFTWARE  LIFE  CYCLE  MANAGEMENT  WKSHOP  0033 

SOFTWARE  ENGINEERING  NOTES  (ACM  SIGSOFT)  0032 

IEEE  SPECTRUM  0001 

PROCEEDINGS,  1969  ANN.  SYMPOSIUM  ON  RELIABILITY  0001 

PROCEEDINGS, 1970  ANN. SYMPOSIUM  ON  RELIABILITY  0001 

CONF.  ON  SPECIFICATIONS  OF  RELIABLE  SOFTWARE, 1979  0019 

2ND  SUMMER  SOFTWARE  ENGINEERING  WORKSHOP  0011 

3RD  SUMMER  SOFTWARE  ENGINEERING  WORKSHOP  0012 

WKSHOP  ON  SOFTWARE  TESTING  &  TEST  DOCUMENTATION  DEC '78  0022 

SOFTWARE  -  PRACTICE  AND  EXPERIENCE  0016 

TOOLS  FOR  EMBEDDED  COMPUTING  SYSTEMS  SOFTWARE  0030 

IEEE  TRANSACTIONS  ON  COMPUTERS  0001 

IEEE  TRANSACTIONS  ON  RELIABILITY  0002 

IEEE  TRANSACTIONS  ON  SOFTWARE  ENGINEERING  0178 

TUTORIAL:  SOFTWARE  MANAGEMENT  0001 


NOTE:  THE  TOTAL  DOCUMENTS  IN  THIS  TABLE  WILL  BE  GREATER  THAN  THE  SUM  OF 
THE  PAPERS  AND  JOURNAL  ARTICLES  IN  THE  PRECEEDING  TABLE.  THE  CAUSE 
IS  THAT  SEVERAL  PAPERS  HAVE  APPEARED  IN  MORE  THAN  ONE  PUBLICATION;  TO 
PROVIDE  ALTERNATE  SOURCES  FOR  OUR  USERS,  WE  HAVE  INCLUDED  REFERENCES 
TO  ALL  SOURCES  OF  WHICH  WE  ARE  AWARE. 
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5.2.1  The  DACS  Thesaurus 


In  order  to  index  the  collection,  it  was  necessary  to,  first,  construct  a 
thesaurus  of  terms  which  could  be  used  for  both  indexing  and  retrieval  purposes. 
This  thesaurus  has  been  named  the  DACS  Software  Engineering  Thesaurus  and  is 
composed  of  475  keywords  or  terms  arranged  into  50  groups  of  closely  related 
keywords  headed  by  a  cluster  term.  A  listing  of  the  50  cluster  terms  is  con¬ 
tained  in  table  5.2-III. 

Appendix  B  contains  a  listing  of  all  475  keywords  in  the  DACS  Thesaurus 
together  with  the  number  of  docunents  to  which  each  keyword  has  been  assigned. 

There  were  8164  keyword  assignments  made  to  1346  online  document  descrip¬ 
tions,  yielding  an  average  of  six  keywords  per  document. 

5. 2. 1.1  Description  of  Keyword  Clusters 

Table  5.2-IV,  contained  on  the  following  pages,  gives  for  each  cluster 
term,  the  number  of  terms  in  that  cluster,  a  description  of  terms  in  that 
cluster,  a  description  of  how  the  terms  are  used  in  indexing  and/or  a  descrip¬ 
tion  of  the  type  of  information  which  a  user  could  expect  to  retrieve  when 
searching  the  bibliographic  datafiles  on  keywords  in  that  cluster. 

5.2.2  Database  Implementation  and  Retrieval 

The  DACS  software  engineering  bibliographic  datafiles  are  implemented  to 
provide  rapid  retrieval  of  individual  document  citations.  The  datafiles  are 
accessed  through  the  Management  Data  Query  System  (MDQS)  implemented  on  the 
Honeywell  6180  computer.  This  database  management  system  allows  searching  on 
any  of  the  items  in  a  bibliographic  citation  and/or  assigned  keywords  and  out¬ 
puts  bibliographies  of  documents  retrieved  during  the  searching  process. 
Searching  may  be  done  on  line  or  in  batch  mode. 

The  retrieval  language  for  MDQS  is  procedural ly  oriented  and  syntactically 
demanding  for  all  but  the  simplest  retrieves  or  where  any  degree  of  formatting 
of  output  is  needed.  To  alleviate  the  problems  inherent  in  such  a  system,  a 
series  of  parameterized  queries  have  been  written.  These  queries  are  maintained 
online  and  can  be  utilized  by  non -prog ramming  personnel.  The  person  running  a 
search  need  only  log  on,  type  the  letters  MDQ  and  the  name  of  the  file  contain¬ 
ing  the  query,  and  then  type  "RUN."  The  MDQS  system  will  request  three  para¬ 
meters,  (the  topic  of  the  search,  the  search  strategy,  and  the  retrieval  quali¬ 
fiers),  actuate  the  retrieval  and  direct  the  resulting  bibliography  to  an  off¬ 
line  printer.  At  the  present  time,  a  DACS  information  specialist  or  the  user 
supplies  the  parameters,  and  the  search  is  actually  run  by  clerical  personnel. 

5.2.3  Document  Processing 

Input  processing  procedures  for  the  software  engineering  documents  have 
been  established  to  assure  that  the  documents  are  relevant  to  the  Center  and 
that  after  inclusion  into  the  bibliographic  collection  they  are  easily  retriev¬ 
able.  The  major  activities  are  log-in,  code,  index,  verify,  keypunch,  and  load. 
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TABLE  5. 2-1 II  CLUSTER  TERMS 


APPLICATIONS 

ARCHITECTURE 

ARTIFICIAL  INTELLIGENCE 

CERTIFICATION 

COMPLEXITY 

CONVERSIONS 

COST 

CORRECTNESS  PROOFS 
DATA 

DEBUGGING 

DOCUMENTATION 

DISTRIBUTED  PROCESSING 

DESIGN 

DEVELOPMENT 

EDUCATION 

ERRORS 

FAILURES 

FAULT 

FUNCTIONS 

HIERARCHY 

IMPLEMENTATION 

LANGUAGES 

MICRO  COMPUTERS 

MODELS 

MEASUREMENTS 

MANAGEMENT 


MINICOMPUTERS 

MODULES 

MODELLING  AND  SIMULATION  TOOLS 

MAINTENANCE 

OPERATING  SYSTEMS 

PERFORMANCE 

PROGRAMS 

PROGRAMMING 

PROCEDURES 

QUALITY 

RELIABILITY 

REQUIREMENTS 

SECURITY 

SOFTWARE  ENGINEERING 

SPECIFICATIONS 

SUPPORT  SOFTWARE 

STANDARDS 

SOFTWARE 

STANDARDIZATION 

TRI-SERVICE 

TESTING 

VERIFICATION 

VALIDATION 

VIRTUAL  MACHINES 
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TABLE  5. 2- IV  DESCRIPTION  OF  KEYWORD  CLUSTERS 


CUSTER  TERM 


APP'.K 

DAT  I ‘DNS 

i  155  <eyworasz 

ARCHITECTURE 

I  Keywords 

w:?: 

:c;al  intelligence 

Keyword) 

DERTH 

:I  CAT  ION  .1  <eyword) 

: 

COMPLEX* 7V 

7  keywords) 

■D3NVE! 

1SD0NS 

A  xeyworasl 

COST 

,'7  keywords; 

:0RRE( 

t  / 

DT'IESS  ’ROOFS 
t  keywords; 

DATA 

10  xeyworqs , 

DE3UGI 

SING  I 

:  <eyworas . 

DESCRIPTrON 


Most  of  these  terms  are  not  sDecific  to  computer  software  or  hardware, 
out  indicate  that  a  document  indexed  using  one  of  tnese  terms  describes 
the  development  or  use  of  software  T'or  a  specific  aDDlication. 


These  terms  reference  documents  related  to  either  software  architec¬ 
ture  or  to  system  architecture  which  'ncluaes  botn  naruware  ana  soft¬ 
ware  architecture. 


"his  term  is  useo  to  i  naex  documents  -elating  to  the  use  of  software 
or  the  aevelooment  of  software  which  aooears  to  reason,  learn,  ana 
improve  itself. 


'his  term  is  usea  to  index  documents  relating  to  any  oarts  of  the 
certification  process  -'or  software.  This  term  may  also  oe  used  as  a 
aart  of  a  ooolean  -etr-eve  with  testing,  support  software,  etc.,  to 
retrieve  on  a  narrower  oasis. 


These  cerms  are  used  to  maex  documents  relating  to  program  complexity, 
programming  language  complexity  and  measurement  df  complexity. 


*hes«  terms  are  usea  to  inaex  documents  referring  to  costs,  method, 
or  aids  for  conversions  df  existing  software  from  one  language  to  an¬ 
other  language  or  from  one  oaraware/software  configuration  to  another. 

These  terms  are  used  to  inaex  ana  retrieve  documents  relating  to  the 
cost  associated  with  any  activity  during  any  ohase  of  the  software 
life  cycle.  This  duster  also  contains  terms  relating  to  oroductivity 
and  to  management  cost  data. 


This  grouo  of  terms  is  used  to  index  ana  retrieve  documents  relating 
to  the  development  or  use  of  correctness  proofs,  by  either  automated 
or  manual  means.  The  terms  termination  proof  and  synodic  execution 
were  also  assigned  to  this  cluster. 


These  terms  reference  sources  of  oata.  means  of  collecting  or  acquir¬ 
ing  data,  and  validation  of  data.  The  data  referenced  relates  to  soft¬ 
ware  aevelooment,  costs,  maintenance  or  reliability. 


These  terms  reference  documents  which  refer  to  methods  for  or  exoeri- 
encas  in  locating  and  correcting  faults  whose  existence  ’S  xnown. 
Documents  indexed  under  these  terms  will  soecifically  refer  to  the 
process  as  "debugging.  "  Other  material  relating  to  the  same  Drocess 
are  indexed  using  various  cents  ‘rom  the  Error  and  Fault  clusters. 


'hese  terms  reference  documents  dealing  with  documentation  by  either 
manual  or  automated  means.  Documents  -naexed  may  re-ar  to  the  Drocess 
of  producing  documentation  pr  to  the  resulting  product  documentation. 
To  retrieve  documents  relating  to  documentation  produced  dur-ng  a  par¬ 
ticular  phase  af  the  software  life  cycle  pr  ‘or  a  particular  puroose, 
,se  a  term  from  -he  documentation  -luster  as  one  term  in  a  boolean 
search,  e.g..  Documentation  and  Desion. 

This  term  references  pr-'-ari1/  documents  relati-g  to  the  development 
of  distr-buted  or  oaral’el  processing  systems,  "he  collection  nas 
very  li-.t’e  ;n  this  area. 


TABLE  5. 2-1 V  con't 


!  OESIGN  (28  keywords) 

These  terms  reference  documents  which  relate  experiences  using  various 
design  tools  or  methodologies,  which  describe  a  particular  design 
tool  or  methodology,  which  describe  particular  activities  occurring 
during  the  design  phase  of  the  software  life  cycle,  or  which  describe 
a  completed  design  for  a  software  product  or  component. 

. 

DEVELOPMENT  (!1  Keywords) 

These  terms  reference  documents  relating  to  the  developmental  process, 
developmental  methodologies  and  developmental  tools.  Documents  relat- 
ing  to  all  of  the  phases  in  the  software  development  cycle  will  be 
indexed  using  a  term  from  this  cluster;  if  the  treatment  is  extensive 
for  each  phase,  it  will  also  be  indexed  using  keywords  which  apply  to 
chat  phase. 

EDUCATION  , 2  Keywords ) 

These  terms  refer  to  formal  and  informal  software  engineering  educa¬ 
tional  processes  and  curricula  including  technology  transfer. 

ESTOPS  '.21  Keywords) 

These  terms  are  used  to  index  and  retrieve  dociments  relating  to  error 
analysis,  detection,  correction,  and  prevention.  Documents  relating 
to  error  data,  and  error  categories  are  also  indexed  using  terms  from 
this  group. 

FAILURES  (6  keywords) 

These  terms  reference  documents  relating  to  failure  rates,  failure 
ratios,  failure  categorization  and  failure  data. 

FAULTS  ( 1 1  keywords ) 

These  terms  reference  docianents  relating  to  both  faults  and  fault- 
tolerance. 

FUNCTIONS  (1  keyword) 

This  term  is  used  to  index  or  retrieve  documents  which  refer  to 
development,  design,  or  programming  approaches  wnicn  view  a  software 
system  component,  or  module  in  terms  of  its  reouired  or  intended 
function. 

HIERARCHY  (3  keywords) 

These  terms  are  used  primarily  as  a  cross  reference  for  boolean  searches, 
e.o.  Hierarchy  and  Desion 

IMPLEMENTATION  (2  keywords) 

These  terms  reference  documents  which  are  concerned  with  the  methods 
and  processes  by  which  a  design  is  transformed  into  a  software  product. 
Documents  concerned  with  the  implementation  phase  of  the  software  life 
cycle  are  indexed  using  these  terms. 

LANGUAGES  (45  keywords) 

These  terms  are  used  to  Index  documents  evaluating  or  comparing  two  or 
more  languages,  or  evaluating  or  describing  particular  features  or 
applications  of  a  given  language.  The  terms  reference  various  program¬ 
ming  languages,  design  languages,  requirements  languages  and  specifica¬ 
tion  languages. 

MICROCOMPUTERS  (4  keywords) 

These  terms  are  used  to  refer  to  the  design  or  development  of  software 
for  microcomputers  or  to  the  use  of  microcomputers  in  a  particular 
application.  These  terms  may  be  used  as  one  term  in  a  boolean  search 
to  retrieve  information  on  a  particular  application  of  microcomputers. 

MOOELS  (27  keywords) 

These  terms  reference  documents  describing  the  use,  validity  or 
development  of  models.  Models  include  reliability,  availability,  and 
error  models.  Documents  are  also  fnoexed  on  the  mathematical  metho¬ 
dology  which  is  used  to  derive  or  which  is  the  form  assumed  by  a  parti¬ 
cular  model . 

MEASUREMENTS  (1  keyword) 

Used  primarily  as  a  cross  reference  keyword  to  allow  retrieval  of 
information  about  measures  or  measurement  techniaues  developed  for  a 
oartlcular  aoolicatlon.  I.e..  Maintainability  and  Measurements. 
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TABLE  5. 2-1 V  con't 


MANAGEMENT  ( 1 !  keywords ) 


MINICOMPUTERS  (1  keyword) 


MODULES  (7  keywords) 


MODELLING  ANO  SIMULATION 
TOOLS  (3  keywords) 


MAINTENANCE  (11  Keywords) 


OPERATING  SYSTEMS 
(17  keywords) 


PERFORMANCE  (3  keywords) 


PROGRAMS  (11  keywords) 


PROGRAMING  (25  keywords) 


PROCEDURES  (1  keyword) 


QUALITY  (29  keywords) 


Terms  used  to  reference  documents  referring  to  management  aspects  of 
software  engineering. 


May  refer  to  the  design  or  development  of  software  for  a  minicomputer 
system,  or  to  the  use  of  minicomputers  in  a  software  development  pro¬ 
ject  (e.g- .  as  an  emulator  or  simulator),  or  to  a  analysis  of  mini¬ 
computers  vs  mainframe  computers  as  components  of  a  hardware/software 
system.  Best  use  is  as  one  term  in  a  ooolean  search  to  retrieve  infor¬ 
mation  on  particular  uses  of  minicomputers. 


This  cluster  contains  a  more  loosely  related  group  of  adjectival  and 
nounal  keywords  used  to  reference  documents  relating  to  software 
designed  and  implemented  in  a  modular  fashion  or  to  the  modules  in 
software  product. 


These  keywords  reference  docunents  concerned  with  software  tools  used 
for  trade-off  studies  and  to  investigate  particular  abstractions  and 
approaches  for  system  design. 


These  terms  reference  documents  concerned  with  the  actities  occurring 
during  the  operations  and  maintenance  phase  of  the  software  life  cycle. 


Terms  in  this  cluster  reference  documents  concerned  with  particular 
aspects  of  operating  systems,  with  design  and  development  of  entire 
operating  systems,  or  with  evaluation  of  a  particular  operating  system. 


These  terms  reference  documents  concerned  with  system  performance, 
either  as  a  whole,  or  with  particular  performance  factors. 


This  cluster  contains  I  mixed  bag  of  keywords  referencing  documents 
concerned  with  attributes  of  programs.  Four  of  the  keywords  in  this 
cluster  reference  program  library  systems. 


Terms  in  this  cluster  reference  documents  concerned  with  prog ramming 
methods  and  techniques  and  programming  for  particular  applications. 
The  terms  usual ly  described  as  'modem  programming  practices"  are  also 
included  in  this  group. 


Procedures  is  a  term  used  ambiguously  in  the  literature  to  refer  to 
functions  performed  by  an  operating  system,  as  a  quasi-synonym  for  a 
module  or  its  output  or  as  a  control  structure.  Best  used  in  a  boolean 
search  combined  with  another  term  so  as  to  narrow  the  retrivals  to 
reflect  the  user's  information  need.  i.e. .  Operating  Systems  and 
Procedures. 


This  cluster  contains  the  terms  which  reference  documents  concerned 
with  the  quality  attributes  or  "ilities."  It  also  contains  the  terms 
related  to  software  quality  assurance. 


RELIABILITY  (14  keywords) 

Conceptually,  reliability  belonqs  in  the  Quality  cluster;  however, 
because  there  is  so  much  literature  on  the  various  aspects  of,  methods 
for  achieving,  controversy  concerning,  and  means  for  measuring  soft¬ 
ware  reliability;  reliability  and  its  related  keywords  have  been  group¬ 
ed  into  a  second  cluster. 

REQUIREMENTS  (6  keywords) 

Terms  which  reference  documents  relating  to  activities  performed  or 
processes  occurring  during  the  requirements  onase  of  the  software 

life  cycle. 
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TABLE  5. 2-1 V  con‘t 


SECURITY  (11  keywords ) 


The  keywords  in  this  cluster  reference  documents  concerned  with  both 
system  access  security,  the  security  of  date  resident  in  tne  system, 
and  the  privacy  concerns  of  entitles  about  whom  (which)  data  is  con¬ 
tained  in  the  system. 


SOFTWARE  ENGINEERING 
(16  keywords  1 


The  keywords  in  this  cluster  reference  documents  concerned  with  soft¬ 
ware  engineering  tools,  techniques,  and  aids;  economic  and  managerial 
aspects  of  software  engineering;  and  general  background  information  on 
software  engineering  as  a  discioline. 


SPECIFICATIONS  (9  keywords) 


Terms  which  reference  docunents  relating  to  activities  performed  or 
processes  occurring  during  the  specifications  phase  of  the  software 
life  cycle. 


SUPPORT  SOFTWARE 
1 1 0  keywords ) 


This  cluster  contains  the  cluster  term,  support  software,  and  nine  suO- 
terms.  The  subterms  refer  to  types  or  classes  of  support  software 
e.g.,  compilers,  editors,  interpreters,  ’he  cluster  term  is  useo  most 
often  to  Index  docunents  related  to  software  tools  and  used  in  a 
boolean  search  to  retrieve  documents  relating  to  a  particular  tool 
type,  e.g.,  Support  Software  and  Testing  would  retrieve  those  docu¬ 
ments  containing  information  about  software  test  tools. 


STANDARDS  (1  keyword) 


Used  in  indexing  primarily  as  a  cross  reference  term  to  provide  for 
boolean  searches  to  retrieve  information  on  the  different  types  of 
standards. 


The  cluster  term  alone  is  never  used  in  indexing.  It  provides  a  home 
for  two  other  terms.  One,  software  life  cycle  is  used  to  retrieve  docu¬ 
ments  which  give  overviews  pr  general  treatments  of  the  software  life 
cycle.  The  second  term,  software  ohyslcs,  is  a  controversial  term, 
with  different  meanings  in  different  camps  and  serious  question  about 
whether  either  meaning  is  valid  for  software  engineering.  The  term  is 
included  in  the  DACS  Thesaurus  because  it  is  in  use  and  because  the 
concepts  referenced  by  it  are  of  concern  to  the  software  engineering 
communl ty. 


STANDARDIZATION  (I  keyword) 


Used  to  refer  to  efforts  or  proposals  for  standards  development, 
used  in  a  boolean  search  for  retrievals. 


TRI-SERVICE  (2  keywords) 


Used  to  reference  docunents  relating  to  joint  U.S.  Army,  u.S.  Navy, 
U.S.  Air  Force  efforts  or  projects  of  interest  to  tne  software  engi¬ 
neering  conmunity. 


TESTING  (28  keywords) 


Used  to  refer  to  various  levels  of  testing,  i.e. ,  module,  program, 
system  testing;  to  various  testing  techniques;  and  to  test  tools  of 
various  types. 


VERIFICATION  (8  keywords) 


Used  to  reference  docunents  containing  information  on  verification 
activities  performed  throughout  or  during  any  phase  of  the  software 
development  cycle. 


VALIDATION  (3  keywords) 


Used  to  reference  docunents  containing  information  on  the  validation 
process. 


VIRTUAL  MACHINES  (7  keywords) 


The  terms  in  this  cluster  reference  docunents  containing  information 
relating  to  abstract  machines,  user- interactive  systems  and  multipro¬ 
gramming  for  such  systems. 


Log-In 


This  activity  begins  the  document- input  process.  The  document  is  tested 
for  uniqueness  of  author  and  title  against  the  current  on-line  author  catalog 

and  the  current  in-process  author  catalog.  If  the  document  is  a  possible  dup¬ 

licate;  it  is  compared  to  the  document  already  in  the  collection;  if  an  exact 
duplicate,  it  is  marked  as  copy  two,  three,  or  four  etc.  and  filed,  if  a  dupli¬ 
cate  from  a  different  source,  the  new  document  is  marked  as  copy  two,  three,  or 
four,  etc.  and  an  entry  is  made  on  an  addenda  sheet  for  keypunching  to  reflect 
the  additional  source. 

If  the  document  is  unique,  a  document  accession  number  is  assigned  and  the 
accession  number,  author  and  title  entered  on  the  DACSLOG  data  sheet  for  entry 

at  the  end  of  the  day  in  the  in-process  author  catalog.  After  entry  on  the 

DACSLOG  data  sheet,  a  document  description  form  is  prepared  which  contains  all 
bibliographic  information  related  to  the  document.  At  this  time,  all  coded 
data  (journal  codes,  availability  codes,  document- type  codes,  author-organiza¬ 
tion  codes  and  contract-sponsor  codes)  are  looked  up  in  the  tables  (or  created 
and  added  to  the  tables)  and  entered  on  the  document  description  form.  After 
LOG  IN,  either  coding  or  indexing  may  be  the  next  step. 

Code 


The  information  contained  on  the  document  description  form  is  transcribed 
onto  the  coding  sheets  for  subsequent  keypunching. 

Index 


Descriptors  (keywords)  are  selected  from  the  DACS  Software  Engineering 
Thesaurus.  The  keyword  codes  selected  are  entered  on  the  Document  Description 
Form  and  also  on  the  coding  sheets  if  coding  of  bibliographic  information  has 
already  been  done.  If  coding  has  not  yet  been  completed,  the  keyword  codes 
are  written  on  the  document  description  form  to  be  coded  when  the  rest  of  the 
bibliographic  information  is  entered. 

Verify 

Completed  coding  sheets  are  reviewed  for  legibility  of  all  characters, 
conformance  to  coding  conventions,  and  accuracy  of  transcription  from  the  docu¬ 
ment  description  forms. 

Keypunch 

The  verified  coding  sheets  are  sent  to  a  keypunching  service  in  batches  of 
50  to  100  sets  of  3  or  4  sheets.  Each  set  represents  one  document.  The  infor¬ 
mation  on  the  coding  sheets  is  transferred  to  punch  cards  and  verified. 

Load 


The  punched  cards  are  read  into  a  BCD  file,  sorted,  converted  to  ASCII 
format,  and  the  card  images  listed  in  hard  copy.  The  listing  is  skimmed  for 
obvious  inconsistencies,  EBCDIC-ASCII  incompatabilities  are  edited,  errors  are 
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edited,  the  file  is  reconverted  to  BCD  and  written  to  the  bibliographic  data¬ 
files  by  invoking  the  Indexed  Sequential  Processor.  At  this  step,  the  six 
lookup  tables  for  coded  data  are  updated  as  necessary.  When  the  new 
bibliographic  data  has  been  loaded,  a  set  of  procedures  which  locates  unmatched 
coded  data  are  run  as  a  final  check.  Any  incorrect  codes  are  noted  in  the 
source  file  for  the  next  file  maintenance  session. 


5.3  The  DACS  Glossary 


There  were  two  objectives  in  compiling  the  DACS  Glossary.  The  first 
objective  was  to  record  the  terminology  currently  in  use.  The  second  was  to 
provide  users  of  the  DACS  software  engineering  bibliographic  services,  a  means 
of  ensuring  that  they  are  using  terms  from  the  DACS  Thesaurus  for  their  retrie¬ 
vals  in  the  same  way  as  the  terms  were  used  for  indexing.  A  closely  related 
objective  is  to  promote  consistency  in  indexing  new  documents  for  the 
collection. 


The  first  version  of  the  DACS  Glossary  was  published  by  the  DACS  in 
October  1979.  Most  of  the  terms  and  definitions  were  compiled  from  the  software 
engineering  literature.  This  DACS  Glossary  contains  1123  terms,  1100  of  which 
have  one  or  more  definitions  and  23  of  which  are  cross-references  to  other 
terms . 


The  glossary  is  now  in  a  maintenance  and  refinement  phase.  A  few  new 
definitions  have  been  added,  and  the  definitions  presently  in  the  glossary  are 
being  edited  for  consistent  format,  removal  of  inaccuracies  and  removal  of  dif¬ 
ferent  terms  which  name  the  same  concept.  The  work  with  the  IEEE  Terminology 
Task  Group  is  providing  a  major  input  to  this  necessary  refinement,  as  discussed 
in  the  next  section. 


Terminology  Task  Group  Coordination 


We  have  taken  an  active  role  in  the  IEEE  Terminology  Task  Group.* 

Shirley  Gloss-Soler,  of  IITRI,  is  chairperson  of  this  task  group.  In  addition, 
a  draft  version  of  the  DACS  Glossary  was  provided  to  this  group  and  was  used 
as  a  base  from  which  candidate  terms  were  selected  for  a  working  draft  of  the 
second  version  of  their  Software  Engineering  Terminology  (SET). 


The  task  group  presently  has  112  members  from  government,  industry,,  and 
the  universities.  Approximately  70  are  active  participants.  To  facilitate 
coordination,  a  steering  committee  of  11  people  and  10  subgroups  was  formed. 
Table  5.3-1  provides  information  on  the  coordination  meetings  held  by  this 
steering  committee. 


Their  draft  terminology,  as  of  1  February  1980,  contains  858  terms.  The 
terminology  document  is  scheduled  to  be  completed  April  1981. 


♦Terminology  Task  Group,  Subcommittee  on  Software  Engineering  Standards, 
Technical  Committee  on  Software  Engineering,  IEEE  Computer  Society. 
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TABLE  5.3-1 

MEETINGS  OF  STEERING  COMMITTEE  OF  TERMINOLOGY  TASK  GROUP  (TTG) 
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SECTION  VI 


TASK  8  PRODUCTS  AND  SERVICES  PREPARATION  AND  DISTRIBUTION 


6.1  Introduction 

This  section  summarizes  the  results  of  the  tasks  to  produce  and  distribute 
the  DACS  products  and  services.  A  characterization  of  these  products  and 
services  is  discussed,  along  with  summarized  quantitative  information  on  re¬ 
quests  processed.  More  detailed  data  on  the  number  and  type  of  requests  pro¬ 
cessed  is  included  in  Section  7.2,  Improving  Center  Effectiveness. 

6.2  Data  Subsets 


The  purpose  of  this  task  is  to  distribute  subsets  of  the  DACS  database  (as 
discussed  in  Section  3.2  of  the  report)  to  aid  in  research  efforts  that  require 
productivity,  cost,  complexity,  error,  and  change  data.  These  datasets  are 
being  used  to  validate  and  refine  software  reliability,  maintainability,  and 
software  productivity  models  and  methods.  The  availability  of  the  RADC  Produc¬ 
tivity  Database  was  announced  in  the  June  1979  DACS  Newsletter.  As  of 
1  February,  32  copies  of  this  dataset  had  been  distributed  in  hard  copy  report 
format  or  on  magnetic  tape. 

To  facilitate  distribution  of  this  dataset,  descriptive  literature  on 
ordering  this  dataset  is  provided  to  the  user  and  contains  a  check-off  list  for 
specifying  the  sorted  order  of  a  computer  listing  and  magnetic  tape  requirements. 
This  data  subset  is  distributed  along  with  a  data  dictionary  describing  the  fJ',ta 
elements  and  a  copy  of  the  draft  report  entitled  "Software  Data  Collection  and 
Analysis"  which  presents  a  preliminary  analysis  of  the  dataset. 

The  availability  of  the  Software  Reliability  Database  was  announced  in  the 
December  1979  DACS  Newsletter.  The  report  entitled  "Software  Reliability  Data" 
has  been  compiled  and  contains  descriptions  of  the  data  and  the  characteristics 
of  the  software  systems,  listings  of  the  data,  and  ordering  information  for 
obtaining  a  copy  of  the  data  on  magnetic  tape.  As  of  1  February  1980,  requests 
for  aoproximately  40  copies  of  this  database  had  been  received. 

The  NASA  SEL  Database  is  presently  being  studied -to  determine  the  options 
for  subsetting  and  disseminating  this  information  in  a  usable  form. 

6.3  Data  Compendi urns 

The  production  of  a  software  engineering  data  compendium  was  initiated  dur¬ 
ing  this  reporting  period.  The  purpose  of  this  compedium  is  to  profile  the  soft¬ 
ware  engineering  data  that  has  been  collected  by  the  NASA  Goddard  Software  Engi¬ 
neering  Laboratory  during  software  development  projects  at  that  facility. 

Each  software  development  project  will  be  described  in  terms  of  its  purpose, 
size,  life  cycle  dates  and  development  methodologies,  tools  and  techniques. 

Data  from  computer  run  analysis  forms  are  used  to  profile  the  purposes  and  re¬ 
sults  of  the  projects'  computer  runs.  From  change  report  forms,  the  data 
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elements  such  as  types  of  errors  and  effort  for  error  changes  are  profiled. 
Several  resource  expenditure  forms  are  used  to  produce  project  profiles  regard¬ 
ing  number  of  computer  runs,  computer  time  and  manpower  hours  by  task. 

Several  other  features  are  included  in  the  compendium  to  enable  potential 
users  of  the  data  to  determine  the  relevance  of  the  data  to  their  applications. 
Included  among  the  features  are  a  data  dictionary  and  an  evaluation  of  the  com¬ 
pleteness  of  the  dat~;  by  project  and  data  type. 

6.4  Technical  Reports 

Two  state-of-the-art  reports  were  produced  during  this  reporting  period. 

The  first  report,  enti tied"  "Quanti tati ve  Software  Models,"  was  compiled  from 
the  Data  Parameters  Report  and  is  a  compendium  of  attributes  of  software  cost/ 
productivity,  reliability/error  analysis,  and  complexity  models.  The  availabil¬ 
ity  of  this  software  engineering  research  review  was  announced  in  the  March  1979 
DACS  Newsletter.  As  of  1  February  1980,  452  copies  of  this  report  had  been  dis¬ 
tributed  to  the  software  technology  community. 

The  second  report  is  entitled  "Software  Maintenance  Technology"  and  pro¬ 
vides  a  synthesis  of  the  information  on  software  maintenance  topics  the t  exists 
in  current  articles,  papers,  reports,  and  books.  This  maintenance  technology 
report  will  be  distributed  as  a  RADC  Technical  Report.  The  availability  will 
be  announced  in  the  March  1980  DACS  Newsletter. 

This  state-of-the-art  report  presents  a  practical  guide  on  software  main¬ 
tenance  tools  in  terms  of  categorization,  potential  use  and  benefit,  and  experi¬ 
ence  in  using  the  specific  tool.  This  report  defines  a  set  of  maintenance  tech¬ 
nology  functions  including  configuration  control;  operations  monitoring/evalua¬ 
tion;  redesign;  code  production  and  analysis;  verification  and  validation;  test¬ 
ing  and  integration;  and  documentation.  Forty  maintenance  tools  and  techniques 
are  defined  for  each  technology  function  and  an  indication  of  how  they  relate  to 
0  &  M  activities  is  provided.  One  of  the  major  findings  of  this  comprehensive 
report  is  that  there  has  been  very  little  determination  of  the  effectiveness 
of  these  tools  and  techniques  on  the  0  &  M  phase,  (reference  14). 

6.5  Technical  Inquiries 

Technical  inquiries  to  the  DACS  are  received  and  processed  on  a  daily  basis. 
These  inquiries  are  received  in  many  forms:  by  letter,  telephone  call,  visit, 
or  by  use  of  the  bibliographic  request  form  contained  in  BIB-1.  The  information 
requests  have  ranged  from  the  very  specific  to  general  questions  on  software 
engineering  methodologies.  Table  6.5-1  contains  the  description  of  39  sample 
inquiries  we  have  received.  The  total  number  of  technical  inquiries  received 
during  the  ten-month  period  from  1  April  1979  through  1  February  1980  was  281. 
This  number  does  not  reflect  the  direct  requests  for  DACS  documents  or  to  be 
placed  on  the  Newsletter  mailing  list. 
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A  technical  inquiry  may  be  answered  in  the  following  one  or  more  ways: 


•  A  custom  bibliographic  search  on  the  subject  area  of  interest  is 
performed. 

•  A  preliminary  analysis  of  the  subject  literature  is  made  and  sum¬ 
mary  information  prepared. 

•  A  subset  of  the  DACS  database  is  produced. 

•  Rel event  DACS  literature  is  distributed. 

•  Referrals  to  other  sources  are  provided. 

The  DACS  has  provided  320  custom  bibliographies  to  users.  The  information 
needs  which  prompted  these  searches  have  ranged  from  broad  information  requests, 
whose  result  could  be  described  as  file  dumps,  to  very  narrow  requests  such  as 
"decision  table  programming  in  FORTRAN." 

The  search  profile  below  indicates  the  13  most  popular  requests.  The 
remaining  46  search  topics  were  requested  8  or  less  times  and  included  such 
topics  as  design,  data  col  lection,  modern  programming  practices,  productivity, 
embedded  computers,  and  software  tools. 

Search  Topic  Number  of  Requests 


Costs 

28 

Quality  Assurance 

20 

Rel iabi 1 i ty 

17 

Testing 

16 

Quality  Factors 

15 

Verification/ Validation 

14 

Development 

12 

Conversion 

n 

Documentation 

n 

Errors 

10 

Maintenance 

10 

Management 

10 

Configuration  Management 

9 

6.6  Miscellaneous  Services 


In  addition  to  the  previously  mentioned  DACS  products  and  services,  we 
have  produced  and  distributed  a  summary  of  the  "Second  Minnowbrook  Workshop  on 
Software  Performance  Modeling,"  edited  and  distributed  copies  of  a  paper  by 
Tom  Gilb  entitled  "Controlling  Maintainability  -  A  Quantitative  Approach  for 
Software,"  distributed  bibliographies  of  RADC/IS  Software  Technology  reports, 
and  took  an  active  role  in  organizing  sessions  and  workshops  related  to  software 
technology. 


TABLE  6.5-1  SAMPLE  DACS  TECHNICAL  INQUIRIES 


Researching  certification  of  computer  programs,  equipment,  and 
personnel 

User  needs  specific  information  on  the  processing  and  validation  of 
meterorological  (weather)  data 

We  are  establishing  an  internal  verification  and  validation  group  and 
we  need  data  on  software  errors,  specifically  software  errors  by 
instruction  type 

Require  information  on  code  verification,  i.e.  tools  that  can  be  used 
to  certify  that  minimum  testing  standards  have  been  met 

Need  information  on  manpower  resource  estimation 

Would  like  to  know  about  any  techniques  to  evaluate  technical  proposals 
concerning  software  development,  both  on  a  "stand-alone"  basis  and  a 
hardware  basis  (i.e.,  complete  system) 

This  office  would  appreciate  being  placed  on  your  distribution  list  and 
would  like  further  information  regarding  the  Data  Analysis  Center 
for  Software 

Please  send  information  describing  the  DACS  charter  and  accomplishments 

User  needs  information  on  life  cycle  costing  figures  or  projections  and 
maintenance  costing  data  on  the  benefits  of  distributed  processing 
systems 

Need  information  on  assembly  language  vs  HOL  programmer  productivity 

Custom  bibliography  on  the  verification  and  validation  of  computer 
software 

Information  needed  to  prepare  a  user's  acceptance  criteria  list  for  a 
system  under  development  and  acceptance  testing 

Interested  in  software  life  cycle  cost  models 

Documentation  to  accompany  testing  -  formats  of  test  plans/specifications 
at  various  stages 

We  would  appreciate  your  sending  copies  of  the  various  forms  used  for 
collection  of  software  data  for  analysis 

Am  looking  for  any  information  on  product  safety  -  the  impact  on  safety 
caused  by  software  problems 


TA8LE  6.5-1  con't 


Would  like  data  on  frequency  of  errors  for  real-time  systems 

Would  like  list  of  references  on  software  quality,  reliability,  and 
testing 

Need  to  develop  standards  and  procedures  for  testing  (unit,  acceptance, 
and  system) 

Need  information  on  estimating  manpower  resources  required  for  software 
projects 

Reviews  weapon  systems  status  prior  to  full  scale  production  -  needs 
background  information  on  error  management,  reliability,  and  quality 

3 

Would  like  information  on  estimating  the  size  of  C  I  systems,  functional 
groups,  and  modules 

Doing  a  study  on  life  cycle  cost  models  for  an  Air  Force  human  resources 
lab;  would  like  information  on  life-cycle  cost  models 

Would  like  list  of  references  to  be  used  in  formulating  and  implement¬ 
ing  software  reliability  and  quality  assurance  programs 

Am  a  government  contractor  -  doing  a  large  programming  job.  Need  to  set 
up  internal  standards  for  design,  coding,  programming  and  also 
evaluation  of  completed  programs 

Would  like  information  on  the  use  and  benefits  of  specific  programming 
tools 

Need  information  on  software  fault  detection  as  well  as  fault  tolerant 
operations  and  recovery;  I  am  specifically  interested  in  techniques 
used  to  structure  software  in  order  to  accomplish  the  above 

Need  to  know  the  cost  impact  of  implementing  a  MIL-S-52779  QA  plan, 
especially  interested  in  cost  impact  during  the  development  phase  of 
the  life  cycle 

Would  like  a  list  of  references  on  design  methodologies 

We  are  specifically  interested  in  samples  of  standards  and/or  procedures 
for  collecting  error  (failure)  information  to  be  used  in  procuring 
digital  flight  control  software;  we  do  not  want  it  for  reliability 
estimation  but  to  determine  what  kinds  of  tools  and  techniques  need 
to  be  developed  to  prevent/detect  these  errors 


TABLE  6.5-1  con't 


Am  looking  for  survey  articles  on  software  acquisition  and  cost 
estimation 

We  have  an  urgent  need  for  definitions  of  software  engineering  terms 

We  are  presently  performing  a  tool  survey  and  would  appreciate  your 
assistance  with  regard  to  a  bibliographic  search  of  your  database 
in  conjunction  with  our  efforts 

My  department  has,  as  its  principal  responsibility,  the  quality  assur¬ 
ance  control  and  monitoring  function  on  a  large  SWS  Navigation  Sub¬ 
system;  the  inclusion  of  computer  software  to  this  quality  disci¬ 
pline  on  this  subsystem  has  been  a  recent  addition,  directed  by  the 
Navy;  any  suggestions  that  would  enhance  the  effectiveness  of  our 
quality  assurance  efforts  would  be  appreciated 

Need  practical  information  on  sizing  and  timing  estimation  methods  - 
current  information  requested 

One  of  UNIDO's  (United  Nations  Industrial  Development  Organization) 
problems  is  funding  the  development  of  duplicate  software  packages 
because  minimum  consideration  is  given  to  portability,  maintainability 
and  reliability;  please  send  me  information  on  these  aspects  of 
software 

Involved  in  a  large  USAF  language  conversion  effort  -  Can't  find  any¬ 
thing  in  the  literature 

The  determination  of  factors  which  contribute  to  software  quality  and 
the  application  of  techniques  to  measure  these  factors 

My  present  research  is  concerned  with  development  of  tools  for  predict¬ 
ing  trouble  spots  in  software  development  projects;  what  data  do  you 
have  available  on  predictive  tools  that  operate  during  the  specifi¬ 
cation  phase  and  try  to  predict  how  errors  will  be  distributed  in 
the  final  delivered  products 
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SECTION  VII 


CENTER  EVALUATION  AND  EFFECTIVENESS 


7. 1  Task  9  Center  Evaluation  Mechanisms 

7.1.1  Introduction 

The  purpose  of  this  task  is  to  develop  a  philosophy  and  criteria  for  eval¬ 
uating  the  effectiveness  of  the  DACS.  One  of  the  basic  elements  of  this  task 
is  to  utilize  the  actual  conditions  of  the  day-to-day  operation  of  the  Center. 
That  is,  no  artificial  tests  are  being  implemented  for  collecting  operational 
DACS  data.  Data  is  collected  on  a  day-to-day  basis  during  the  operation  of  the 
DACS  to  account  for  personnel  and  service  related  activities.  The  types  of 
data  collectedare  discussed  in  this  section  of  the  report.  An  evaluation  of  the 
data  is  provided  in  Section  7.2,  Improving  Center  Effectiveness. 

A  questionnai re  was  developed  to  provide  feedback  from  the  user.  The 
results  of  the  survey  are  summarized  in  this  section. 

7.1.2  Center  Acti vi ties 

An  accounting  mechanism  has  been  established  which  provides  the  capability 
to  record  all  resource  expenditures  (personnel  hours  and  cost,  travel  costs,  and 
purchased  materials  and  services)  by  major  DACS  activity.  There  are  20  major 
activities  (ventures)  and  they  are  listed  in  Figure  7.1-1.  This  data  is  report¬ 
ed  on  a  monthly  basis. 

Information  on  the  services  requested  is  recorded  on  the  DACS  Service 
Request  Record  (SVC).  This  record  contains  the  name  and  affilation  of  the  user, 
the  date  requested  and  answered,  the  request  type,  a  narrative  description  of  the 
request,  the  recommended  action,  the  response,  and  the  resources  expended.  Each 
SVC  is  assigned  a  unique  number.  A  User  Profile  Record  (file)  is  maintained 
for  each  organization  and  summarizes  the  requests  of  all  individuals  within  the 
organization.  The  SVC's  are  then  filed  in  their  respective  User  Profile  Folder. 
A  DACS  document  log  is  also  maintained  on  a  continuing  basis  which  records  docu¬ 
ment  distribution  according  to  type,  date,  and  user. 

A  document  processing  log  provides  controls  for  keeping  track  of  the  pro¬ 
cessing  of  software  engineering  documents  for  the  DACS  document  library.  This 
log  records  the  date  the  document  was  assigned  a  unique  document  accession  num¬ 
ber,  the  date  when  the  document  was  coded  and  indexed,  the  date  the  coding  was 
verified,  the  date  the  coding  sheets  were  sent  to  be  keypunched,  and  the  date 
when  the  document  citation  was  entered  into  the  bibliographic  database. 

This  operational  data  is  summarized  and  incorporated  into  the  monthly 
status  report.  The  status  report  contains  the  number  of  requests  processed, 
the  number  and  types  of  products  (documents)  distributed,  the  growth  of  the 
database,  the  number  of  names  on  the  newsletter  mailing  list,  and  the  number  of 
documents  received,  accessioned,  and  entered  into  the  bibliographic  data  files. 
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Task  1  Establish  and  Operate 


A  17  Establish  and  operate  the  Center  -  administrative 

Tasks  2  and  5  Develop  DB  and  Process  Input 

A  20  Database  development  and  maintenance 
A  21  Input  processing  -  data 

Tasks  3  and  4  Data  Analysis  and  Data  Parameters 

A  30  Data  Analysis  Program  (SRR-2  production) 

A  31  BSDS  Analysis 

A  32  Data  Parameters 

Task  6  Current  Awareness  Program 

A  40  Develop  and  expand  general  user  awareness  (newsletters  and 
promotional  material) 

A  41  Preparation  and  participation  in  meetings  and  conferences 
Task  7  STINFO 

A  50  Establish  and  maintain  library 
A  51  Software  Engineering  Glossary 
A  52  Review  research  projects 

Task  8  Products  and  Services 

A  60  Preparation  and  distribution  of  data  subsets 

A  61  Preparation  and  distribution  of  data  compendiums 

A  62  Final  preparation  and  distribution  of  technical  reports  (SRR-1) 
A  63  Final  preparation  and  distribution  of  bibliographic  products 
(BIB-1,  Thesaurus,  Glossary) 

A  64  Technical  and  bibliographic  inquiry  services  (SVC) 

Tasks  9,  10  and  11 


A  70  Evaluate  and  improve  Center 
A  71  Establish  a  cost  recovery  program 

Task  12 

A  80  Computer  program  documentation 


FIGURE  7.1-1  DACS  ACTIVITIES 


7.1.3  User  Survey 
7. 1.3.1  Introduction 


A  questionnaire  was  designed  and  distributed  with  the  September  1980  DACS 
Newsletter  (Figure  7.1-2).  The  survey  contained  questions  concerning  the  type 
of  job,  ranking  of  the  value  of  types  of  information,  an  indication  of  what 
DACS  products  and  services  they  had  used,  the  usefulness  of  these  products  and 
services,  and  general  comment  questions. 

A  summary  of  the  results  of  the  survey  is  contained  in  this  section  of  the 
report.  Then,  in  Section  7.2,  these  results  are  compared  to  the  information 
collected  on  the  operation  of  the  DACS. 

7. 1.3. 2  Summary  of  Results 

The  September  1979  DACS  Newsletter,  which  included  the  survey  question¬ 
naire,  was  sent  to  approximately  2650  individuals  and/or  organizations.  By  the 
end  of  January,  1980,  169  responses  had  been  received— a  response  rate  of  6%. 
This  is  a  very  low  response  rate  and  sample  size  to  provide  any  major  conclu¬ 
sions.  Therefore,  the  analysis  of  the  survey  presented  in  this  report  should 
be  read  with  this  point  in  mind. 

The  results  summarized  below  are  presented,  first,  by  each  individual 
question  and  the  response,  and  subsequently,  by  correlating  response  categories 
of  interest. 

Question  1  -  User  Type 

This  first  question  was  descriptive  in  nature  and  oriented  towards  the 
type  of  job  that  the  recipient  presently  held.  These  job  descriptions  were 
categorized  into  three  major  types.  The  percentage  of  respondees  for  each  major 
job  type  are  listed  below: 


Manager 

35% 

Prog  r amme  r/Analyst 

36% 

Researcher 

29% 

Question  2  -  Rank  of  Information  Type 

This  question  was  included  to  help  determine  what  type  of  information  the 
respondees  felt  was  most  valuable  to  them  in  performing  their  job.  They  were 
asked  to  rank  seven  information  types. 

Figure  7.1-3  illustrates  the  number  of  respondees  who  rated  each  informa¬ 
tion  type  as  a  1,  2  or  3,  where  a  1  represents  the  most  valuable.  As  shown  in 
this  figure,  the  greatest  number  of  respondees  indicated  that  state-of-the-art 
reports  were  considered  most  valuable,  followed  by  information  on  new  software 
engineering  research.  This  figure  also  shows  that  bibliographies  and  raw  data 
are  not  one  of  their  top  three  priority  items. 
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DACS 


User’s  Survey 


To  nelp  us  plan  the  direction  of  the  Center  and  evaluate  its  present  services,  we  need 
/our  feedback.  3y  completing  the  following  survey  you  can  provide  this  feedback  and  make 
the  DACS  more  responsive  to  your  present  and  future  information  needs.  The  survey  takes 
just  a  few  minutes  to  complete  and  is  postage-paid.  Thank  you  for  your  participation. 


1.  3nefly  describe  your  job,  especially  any  software-related  aspects. 


3.  For  your  ;ob,  please  rank  the  value  of  th 
valuable) 

!  )  Information  on  new  s/w  engineering 
research 

!  )  Surveys  of  currenc  technologies, 
state-of-the-art  reports 

(  !  Bibliographies 

(  )  Raw  data  ie.g.  ,  s/w  cost,  failure 
productivity  data) 

3.  What  DACS  products  or  services  have  you 
;  )  Thesaurus/DACS  User ' s  Guide 

(  )  News letter 

(  )  SRR-1,  Quantitative  S/W  Models 
!  )  Data  subsets 

4.  How  useful  were  the  products/services? 

!  I  'Jaeless  (  )  Marginally  useful 

5.  what  products  or  services  should  the  DA' 


following  types  of  information.  (1  «  most 

(  )  Information  on  previous  s/w  engineering 
research 

I  )  Handbook  type  information  (specs, 
guides,  charts,  tables,  etc.) 

(  )  Critical  reviews  of  s/w  engineering 
literature 

used? 

(  )  Custom  Bibliographies 
1  )  Technical  Inquiries 

(  )  _ 

<  i  _ 

(Please  check  one) 

(  )  Fairly  useful  (  )  Veiy  useful 
:S  offer?  ("Blue-sky"  this  one) 


6.  Any  other  comments  on  your  information  requirements  or  DACS  services? 
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FIGURE  7.1-2  USER  SURVEY  FORM 
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of  Responses 


#1  Rating  (most  valuable) 
#2  Rating 
#3  Rating 


FIGURE  7.1-3  INFORMATION  TYPE  -  MOST  VALUABLE 


Question  3  -  Usage  of  DACS  Products  and  Services 


This  question  was  included  in  the  survey  to  determine  what  DACS  products 
and  services  the  respondees  had  used.  One  of  the  more  interesting  results 
shows  that,  even  though  the  questionnaire  was  actually  a  part  of  the  Newsletter, 
only  88%  of  the  respondees  indicated  that  they  had  used  the  Newsletter.  This 
may  be  explained  in  part  by  some  of  the  comments  received  stating  that  this 
September  Newsletter  was  the  first  time  they  had  heard  of  the  Center,  but  they 
were  interested  in  receiving  more  information. 

Listed  below  is  the  percentage  of  respondees  that  indicated  they  had  used 
the- specific  DACS  products  and  services. 


Thesaurus/DACS  User's  Guide  38% 
Newsletter  88% 
SRR-1,  Quantitative  S/W  Models  16% 
Data  Subsets  4% 
Custom  Bibliographies  16% 
Technical  Inquiries  12% 


Question  4  -  Products/Services  Evaluation 

The  respondees,  in  this  question,  were  asked  to  indicate  how  useful  they 
considered  the  DACS  products  and  services.  The  percent  of  respondees  indicat¬ 
ing  the  four  ranking  levels  is  illustrated  in  Figure  7.1-4. 

Although  these  results  are  skewed  to  the  useful  side,  they  were  somewhat 
disappointing  in  light  of  general  comments  we  have  received  about  the  DACS.  To 
provide  some  insight  to  these  rankings,  we  analyzed  the  comments  relating  to 
response  and  DACS  usage.  These  analyses  are  discussed  in  subsequent  paragraphs. 

Questions  5  and  6  -  Needs  and  Comments 

A  majority  of  the  respondees  (70%)  provided  descriptive  comments  on  their 
information  needs  and  their  evaluation  of  the  DACS  products  and  services. 

Some  of  the  comments  on  information  needs  were  a  re-iteration  of  their 
perferences  indicated  in  Question  2;  others  were  specific  technology  related 
needs  including--"information  on  software  QA,  online  access  to  the  database,  a 
bibliography  of  SE  tools,  critical  reviews  of  software  tools,  sponsor  confer¬ 
ences  or  workshops,  standards  activity  clearing  house,"  etc. 

Twenty-two  percent  of  the  respondees  provided  complimentary  comments  on 
the  DACS  including--"keep  up  the  good  work,  I  would  like  to  commend  your  present 
efforts,  a  worthy  cause  -  keep  it  up,  enjoy  it  very  much  -  it  fills  a  universal 
need  of  an  overview,  endorse  the  DACS  philosophy,  I  like  the  current  issue,"  etc. 
As  would  be  expected,  the  majority  of  these  comments  were  provided  by  those 
respondees  who  had  rated  the  Center  as  very  useful  or  fairly  useful. 
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FIGURE  7.1-4  RATING  LEVELS 


Eight  of  the  complimentary  comments  were  from  those  individuals  who  had 
rated  the  products/services  as  marginally  useful,  and  one  who  had  not  provided 
a  rating. 

There  were  also  many  requests  for  clarification  on  the  products  and 
services  of  the  DACS  and  to  be  placed  on  the  newsletter  mailing  list. 

User  Type  vs  Information  Need,  Products  Used,  and  Usefulness  Rating 

There  seemed  to  be  very  little  difference  between  user  type  and  the  answers 
to  questions  2,  3,  and  4.  Some  of  the  more  significant  findings  are  discussed, 
although  statistically  there  is  no  difference  between  the  groups. 

Forty- three  percent  of  the  researchers  rated  information  on  new  software 
engineering  research  as  their  highest  priority  information  need  compared  to 
26%  for  managers  and  22%  for  programmer/analysts.  This  compares  to  40%  of  the 
managers  who  rated  state-of-the-art  reports  as  their  highest  information  need, 
followed  by  31%  for  programmer/analysts,  and  30%  for  researchers. 

More  managers  (97%)  indicated  that  they  had  used  the  Newsletter  than  pro¬ 
grammer/analysts  (79%)  or  researchers  (91%). 

The  greatest  difference  in  the  answers  to  the  usefulness  question  was  in 
the  fairly  useful  categroy.  Fifty-six  percent  of  the  researchers  checked  fairly 
useful,  compared  to  44%  for  managers  and  27%  for  programmer/ analysts.  No 
research er  checked  useless. 
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7.2  Task  10  Improving  Center  Effectiveness 

7.2.1  Introduction 

This  section  of  the  report  provides  an  analysis  of  the  inquiries  processed 
during  Phases  I  and  II,  a  discussion  on  the  processing  of  the  documents  for  the 
software  engineering  bibliographic  database,  and  a  description  of  the  activities 
planned  for  Phase  III. 

7.2.2  User  Inquiries 

7. 2. 2.1  Phase  I 

During  Phase  I  (15  August  1978  to  1  April  1979)  of  the  development  of  the 
Center,  91  technical  and  information  inquiries  were  received  and  processed. 

The  majority  of  these  requests  centered  around  more  information  about  the  Center. 
In  answer  to  these  inquiries,  copies  of  two  papers  were  distributed  entitled 
"An  Introduction  to  the  Data  and  Analysis  Center  for  Software"  and  "An  Introduc¬ 
tion  to  the  DACS  Software  Engineering  Bibliographic  Database." 

We  were  not  well  equiped,  during  this  period,  to  provide  much  technical 
assistance;  although  we  did  produce,  manually,  some  custom  bibliographies  and 
provided  referrals  to  other  sources  of  information.  The  table  below  illustrates 
the  number  of  inquiries  processed  per  month. 

TABLE  7.2-1  INQUIRIES/MONTH  -  PHASE  I 


Month 

Number 

CO 

October 

2 

November 

12 

December 

12 

January 

12 

February 

20 

March 

33 

These  inquiries  do  not  reflect  those  users  who  requested  to  receive  the 
DACS  Newsletter.  As  of  1  April,  there  were  2143  names/organizations  on  the 
Newsletter  mailing  list. 

7. 2. 2. 2  Phase  II 

On  1  April  1979,  we  modified  our  method  for  recording  and  processing  of 
user  requests.  We  classified  the  requests  into  three  basic  categories: 

•  Newsletter  Inquiries 

•  Document  Inquiries 

•  Technical  Inquiries 


43 


V  , 


-ftrA. 


A  Newsletter  Inquiry  was  a  request  to  be  placed  on  the  Newsletter  mailing 
list;  a  Document  Inquiry  was  a  specific  request  for  a  DACS  document  or  docu¬ 
ments;  a  Technical  Inquiry  was  a  request  for  technical  information  which  re¬ 
quired  some  engineering  analysis  to  respond.  These  technical  inquiries  we 
answered  in  a  variety  of  ways,  dependent  upon  the  type  of  request,  the  depth 
and  breadth  of  our  information  base  on  the  subject  area  of  interest,  and  the 
availability  of  engineering  time.  Custom  bibliographic  searches  were  performed, 
analyzed,  and  synthesized;  DACS  documents,  or  subsets  of  the  DACS  database, 
were  distributed,  and/or  the  user  was  referred  to  other  sources  of  information 
or  contacts. 

■Figure  7.2-1  provides  information  on  the  number  of  requests  for  each  cate¬ 
gory  per  month.  The  total' number  of  inquiries  processed  during  this  10-month 
period  (1  April  1979  to  1  February  1980)  was  2162.  This  averages  out  to 
approximately  10  inquiries  per  day.  The  total  number  of  inquiries  by  category 
were: 


Technical 

281 

Newsletter 

669 

Document 

1212 

TOTAL 

2162 

The  percentage  of  Technical  Inquiries  by  institution  type  is  listed  below. 


Government 

32% 

Industrial 

60% 

Academic 

4% 

Foreign 

4% 

This  data  was  compared  to  the  percentages  from  two  DLA  information  analysis 
centers;  Metals  and  Ceramics  Information  Analysis  Center  (MCIC  -  reference  19) 
and  the  Reliability  Analysis  Center  (RAC  -  reference  20). 


Requestor 


Percent  of  Total  Inquiries 


RAC  MCIC 


Government 
Industrial 
Academi c 
Foreign 


1 97o  20% 

76%  74% 

(not  recorded)  3.5% 

14%  2.5% 


One  of  the  major  benefits  of  an  information  center  is  to  provide  rapid 
response  to  requests  for  technical  information.  In  analyzing  the  response  time 
data  for  Phase  II,  we  have  not  processed  requests  in  a  timely  enough  manner. 
Steps  are  being  taken  to  alleviate  these  problems  by  assigning  more  flexible 
responsibilities  so  that  work  loads  for  individual  members  of  the  staff  may  be 
distributed  more  evenly. 
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FIGURE  7.2-1 


-  PHASE  II 


Below  is  a  summary  of  the  number  of  working  days  to  answer  a  request  dur¬ 
ing  this  10-month  period. 

Number  of  Working  Days  Percent  of  Total  Technical  Inquiries 

0-  4  61% 

5-  9  16% 

10-14  8% 

>  15  15% 

During  December  and  January  approximately  90%  were  answered  within  the 
first  five  days  while  in  April  and  November  only  20%  were  answered  in  the  first 
five  days. 

Over  3200  documents  were  distributed  in  response  to  the  1200  document 
requests.  Below  is  a  list  of  the  documents  most  requested. 

Document  Title  Number  Distributed 


Bibliographic  Services,  Custom  Searches 


(BIB- 1 )  817 
DACS  Thesaurus  802 
DACS  Glossary  (GL0S-1)  764 
Quantitative  Software  Models  (SRR-1)  452 
An  Introduction  to  the  Data  and 

Analysis  Center  for  Software  195 


The  respondees  to  the  DACS  User  Survey  indicated  a  preference  for  state- 
of-the-art  reports.  They  also  indicated  a  low  interest  in  custom  bibliographies. 
This  is  somewhat  out  of  line  with  the  statistics  on  the  requests  for  documents, 
in  that,  almost  twice  as  many  requests  for  the  bibliographic  services  guide 
were  received  than  for  the  state-of-the-art  report  on  Quantitative  Software 
Models.  This  may  be  explained  in  part  by  the  fact  that  this  report  is  in  a 
specialized  area. 

Another  mechanism  for  receiving  feedback  from  users  is  to  include  a  cri¬ 
tique  response  form  with  each  document  and/or  technical  inquiry  response. 
However,  the  response  to  this  mechanism  tends  to  be  low,  as  reported  in  refer¬ 
ence  15.  The  Defense  Documentation  Center  reported  on  one  sample  of  critique 
forms  sent  with  4262  documents.  Only  98  of  the  forms  were  returned,  or  2.3 
percent.  They  indicate  that  a  better  rate  is  achieved  only  as  a  result  of 
active  follow-up  by  project  personnel. 

7.2.3  User  Awareness 

User  awareness  is  an  important  aspect  of  center  management  in  that  it  pro¬ 
vides  managers  and  technologists  with  clear  and  available  descriptions  of  the 
products  and  services.  During  this  reporting  period  we  have  disseminated  this 
information  through  the  newsletters,  informational  brochures,  and  participation 
in  conferences  and  workshops  (as  discussed  in  Section  IV). 
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DLA  (as  reported  in  reference  16)  has  distributed  more  than  30,000  copies 
of  their  User's  Guide.  But  still  they  find  that  DoD  personnel  require  informa¬ 
tion  but  do  not  know  that  it  is  available  from  the  various  Centers.  Not 
enough  of  the  leaders  of  the  scientific  and  technical  community  are  aware  of  an 
IAC.  The  DoD  IAC  directors  and  managers  also  believe  that  working  engineers  and 
scientists,  who  may  have  used  IAC  services,  tend  to  forget;  and  need  periodic 
reminders  of  the  availability  and  value  of  IAC  services  (reference  17). 

We  have  found  that  the  response  to  vague  definitions  of  our  products  and 
services  has  been  very  low,  while  the  more  succinct  descriptions  (as  in  the 
December  Newsletter)  produce  a  much  higher  response  rate. 

7.2.4  Information  Processing 

As  reported  in  Section  II,  during  this  reporting  period  we  have  concen¬ 
trated  on  establishing  a  technology  information  base  while  continuing  to  serve 
the  user.  During  Phase  III  we  will  add,  to  this  information  base,  compilations 
and  descriptions  of  software  engineering  research  projects.  However,  we  are 
already  encountering  a  large  backlog  in  the  processing  of  the  software  engi¬ 
neering  literature. 

The  document  processing  described  in  Section  5.2.3  was  developed  and 
refined  during  the  past  year  and  tends  to  work  well.  The  area  that  most  needs 
improvement  is  in  the  amount  of  staff-hours  devoted  to  the  Log-in,  Code,  and 
Index  steps.  As  the  DACS  becomes  better  known,  we  are  receiving  references  to 
technical  and  research  reports  from  our  users  which  are  appropriate  for  the 
collection  and  which  ought  to  be  ordered  and  processed.  The  number  of  journals 
and  conference  proceedings  relating  to  software  engineering  is  also  increasing. 

The  backlog  of  documents  queued  at  each  of  the  first  three  steps  of  the 
document  processing  path  increases  each  month.  For  this  reason,  the  DACS  has 
not  initiated  an  aggressive  document  acquisition  program  -  we  do  not  have  the 
staff  to  process  what  is  on  hand  now;  an  increase  in  acquisitions  without  a 
corresponding  increase  in  staff  would  only  increase  the  backlog. 

An  analysis  was  performed  to  determine  the  staff  time  needed  for  each  pro¬ 
cessing  step  and  the  number  of  documents  backlogged  at  each  step.  This  is 
summarized  below. 


Step 

Best  Time 

Worst  Time 

Avg  Time 

No. 

Backlogged  at  This 
Step 

Log-In 

5  min 

26  min 

16  min 

416  documents 

Code 

5  min 

15  min 

8  min 

220  documents 

Index 

5  min 

20  min 

10  min 

322  documents 

Verify 

2  min 

10  min 

3  min 

36  documents 

Keypunch 

4  days 

1%  weeks 

1  week 

none 

Load 

3  hours 

l  week 

5  hours 

none 

47 


7.2.5  Phase  III 

As  stated  in  Section  II,  Phase  III  of  the  development  of  the  DACS  will 
concentrate  on  providing  more  quality  and  indepth  products  and  services.  The 
following  products  are  planned: 

•  The  Second  Edition  of  the  DACS  Glossary 

•  A  report  summarizing  software  engineering  research  projects 

•  Newsletters  and  Bulletins 

•  Two  Software  Engineering  Bibliographies 

•  Two  Technical  Monographs 

•  Two  Data  Compendiums 

In  addition  we  plan  to  intiate  an  active  data  acquisition  program  and  to 
enhance  our  data  analysis,  synthesis,  and  dissemination  services. 


SECTION  VIII 


CONCLUSIONS  AND  RECOMMENDATIONS 


8. 1  Conclusions 

•  A  solid  base  of  information  on  software  engineering  has  been 
developed. 

•  A  framework  for  processing  and  synthesizing  this  information  has 
been  established. 

•  The  products  and  services  of  the  DACS  are  needed  and  can  make  a 
significant  contribution  to  the  development  and  maintainance  of 
quality  software. 

•  Information  on  software  engineering  is  generally  available  -  but 
is  scattered,  is  in  many  different  forms,  and  is  difficult  to  syn¬ 
thesize  for  practical  use  and  evaluation. 

•  The  objectives  established  in  the  Software  Data  Repository  Study 
are  generally  sound. 

•  There  have  been  many  levels  of  requests  during  this  period  from 
the  very  knowledgeable  to  the  very  inexperienced. 

8.2  Recommendations 

•  Provide  more  indepth  analysis  of  the  DACS  information  and  data  and 
package  these  analysis  results  into  technical  monographs  for  gen¬ 
eral  distribution. 

•  Establish  an  information  base  on  software  engineering  research 
projects. 

•  Produce  more  distinct,  descriptive  literature  on  the  products  and 
services  of  the  DACS. 

•  Initiate  an  active  data  acquisition  program  to  expand  the  database 
for  analysis  and  distribution. 

•  Obtain  additional  resources  to  be  used  for  expanding  the  information 
base  and  for  performing  engineering  analysis. 
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APPENDIX  A 


Partial  List  of  DACS  Users 


UNITED  STATES  CORPORATIONS 

Accutest  Corporation 
Chelmsford,  MA 

Advanced  Systems  Incoterm  Corporation 
Well  si ey  Hills,  MA  . 

Advanced  Technology,  Incorporated  * 
San  Diego,  CA 

The  Aerospace  Corporation  * 

Los  Angeles,  CA 

American  Society  of  Civil  Engineers 
Kensington,  MD 

Analytics 
McLean,  VA 

Argonne  National  Laboratory  * 

Argonne,  IL 

Arthur  Andersen  &  Company 
Chicago,  IL 

Automation  Industry,  Incorporated 
Silver  Spring,  MD 

Bell  Laboratories 
Holmdel ,  NJ 

Bell  Northern  Research  * 

Palo  Alto,  CA 

Bendix  Research  Laboratories  * 
Southfield,  MI 

Boeing  Aerospace  * 

(several  locations) 

Burroughs  Corporation  * 

(several  locations) 


CBS  Publishing  Group 
New  York,  NY 

CTEC  Corporation  * 

Falls  Church,  VA 

Calculon  Corporation  * 

Falls  Church,  VA 

Control  Data  Corporation  * 

(several  locations) 

Cinncinnati  Electronics 
Cinncinnati,  OH 

Computer  Sciences  Corporation  * 
(several  locations) 

Computer  World 
Newton,  MA 

Consolidated  Freightways  * 

Portland,  OR 

Cutler-Hammer 
Long  Island,  NY 

Data  Systems  Technical  Informations  Ctr 
New  Haven,  CT 

John  Deere  Company  * 

Waterloo,  IA 

Digital  Equipment  Corporation 
Maynard,  MA 

Distributed  Systems  Corporation 
Chelmsford,  MA 

Doty  Associates,  Incorporated  * 
Rockville,  MD 
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Multiple  Technical  Inquiries 
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Draper  Laboratory  * 

Cambridge,  HA 

Oouglas  Aircraft  Company  * 

Long  Beach,  CA 

Dynamics  Research  Corporation  * 
Wilmington,  MA 

E-Systems,  Incorporated 
Falls  Church,  VA 

EG&G,  Idaho 
Idaho  Falls,  ID 

Evaluation  Associates 
Bala  Cynwyd,  PA 

Ferguson-Bryan  &  Associates,  Inc. 
Washington,  DC 

Fireman's  Fund  Insurance  Company 
San  Rafael,  CA 

Ford  Motor  Company 
Dearborn,  MI 

Gagliardi  Systems  Group,  (GSG) 
Rome ,  NY  \ 

General  Dynamics  * 

(several  locations) 

General  Electric  Company  * 
(several  locations) 

General  Research  Corporation  * 
(several  locations) 

Grumman  Aerospace  Corporation  * 
Bethpage,  NY 

Harris  Corporation 
Melbourne,  FL 

Health  Products  Research,  Inc. 
Somerville,  NJ 

*  Multiple  Technical  Inquiries 


Honeywell,  Incorporated  * 

(several  locations) 

Hughes  Aircraft  Company  * 

(several  locations)  CA 

I  IT  Research  Institute  * 

(several  locations) 

ITT  Telecommunications  Technical  Ctr 
Stratford,  CT 

International  Business  Machines  * 
(several  locations) 

Litton-Mellonics 
Springfield,  VA 

Lockheed  Georgia  Company 
Marietta,  GA 

Lockheed  Missiles  &  Space  Company,  Inc. 
Sunnyvale,  CA 

Logicon,  Incorporated 
San  Pedro,  CA 

Manufacturing  Data  Systems,  Inc. 

Ann  Arbor,  MI 

Martin-Marietta  Corporation  * 

(several  locations) 

McCabe  &  Associates  * 

Columbia,  MD 

McDonnell  Douglas  Corporation  * 

(several  locations) 

Arthur  G.  McKee  S  Company 
Cleveland,  OH 

Measurement  Concept  Corporation 
Rome,  NY 

Memorex 

Santa  Clara,  CA 
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Mitre  Corporation  ★ 

Bedford,  MA 

NCR  Corporation  * 

San  Diego,  CA 

0A0  Corporation  * 

El  Segundo,  CA 

Peat,  Marwick,  Mitchell  &  Company  * 

New  York,  NY 

Perkin-Elmer  * 

Danbury,  CT 

Planning  Research  Corporation 
McLean,  V A 

Price  Waterhouse  &  Company  * 

New  York,  NY 

Quantitative  Software  Management,  Inc: 
McLean,  VA 

The  Rand  Corporation  * 

Santa  Monica,  CA 

Raytheon  Company  * 

(several  locations) 

Raytheon  Service  Company 
Arlington,  VA 

Research  Triangle  Institute 
Research  Triangle  Park,  NC 

Rockwell  International  * 

Cedar  Rapids,  IA 

RCA  Government  Communications  System 
Camden,  NJ 

SAI  Comsystems  Corporations  * 

San  Diego,  CA 

Sheridan  Oaks  Cybernetics 
Glenview,  IL 

*  Multiple  Technical  Inquiries 


Software  Enterprises  Corporation  * 
Westlake  Village,  CA 

Softech  Incorporated  * 

Waltham,  MA 

Software  Management  Consultants  * 
Torrance,  CA 

Software  Research  Associates 
San  Francisco,  CA 

SoHar  Incorporated 
Los  Angeles,  CA 

Southern  New  England  Telephone  Company* 
New  Haven,  CT 

Sperry  Rand  Corporation  * 

(several  locations) 

Syntex  Corporation 
Palo  Alto,  CA 

System  Development  Corporation  * 

Santa  Monica,  CA 

Systems  and  Applied  Sciences  Corporation 
Riverdale  MD  * 

Systems  Engineering  Laboratories  * 

Fort  Lauderdale,  Florida 

Syl vania,  Incorporated  * 

Needham,  MA 

TRW  * 

Redondo  Beach,  CA 

Teletype  Corporation 
Chicago,  IL 

Texas  Instruments  * 

(several  locations)  TX 

Time  Share  Corporation 
Hanover,  NH 
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Union  Carbide  Company  * 

(several  locations)  NY 

Westinghouse  I  LSD  Division 
Hunt  Valley,  MD 

Xerox  Corporation 
Rochester,  NY 

DEPARTMENT  OF  DEFENSE  . 

Defense  Documentation  Center  * 
Alexandria,  V A 

Defense  Logistics  Agency  * 
Alexandria,  VA 

US  AIR  FORCE 

Air  Force  Data  Services  Center  * 
AFDSC/XMD  Pentagon 
Washington,  DC 

Air  Force  Reserve  * 

SD/CSB  RES.  DET.  1 
Santa  Clara,  CA 

Andrews  Air  Force  Base  * 

AFSC/XRF 
Washington,  DC 

Gunter  Air  Force  Base  * 

Phase  IV  PMO/PGY 
and 

AF  DSDC/SCDP 
Montgomery,  AL 

Hanscom  Air  Force  Base  * 

AFTL  Laboratory 
and 

AFGL/SULLR 

and 

ESO/TOI ,  TOIS ,  TOIT,  TOIQ,  TOIT  Hqs. 
Bedford,  MA 


Kirkland  Air  Force  Base  *' 

HQS  AFCDM/QA 
and 

Air  Force  Testing  Evaluation  Center 
NM 

Langley  Air  Force  Base 
HQS  TAC/ADYS 
VA 

McClellan  Air  Force  Base  * 

Sacramento  Air  Logistics  Center 
Air  Force  Logistics  Command 
and 

SM-ALC/MMM-1 

and 

ACD 

CA 

Maxwell  Air  Force  Base 

ACSC/EDOI 

AL 

Nellis  Air  Force  Base  * 

Tactical  Fighter  Weapons  Center 

SA/M 

NV 

Offutt  Air  Force  Base  * 

HQS  SAC 
ADOS 

and 

HQS  SAC/APXPO 

Global  Weather  Central  /DOY 
Global  Weather  Central  /ADSV 
NB 

Griffiss  Air  Force  Base  * 

RADC  ISIS,  ISCP,  RBC 
Rome ,  NY 

Robbins  Air  Force  Base  * 

WR/ALC/MMECV 

GA 


* 


Multiple  Technical  Inquiries 
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Wright-Patterson  Air  Force  Base  * 
AFLC 

and 

AFAL/AAA-3 

AFFA/ASD 

ASD/YPAA 

AFLC/ACMCE 

OH 

US  ARMY 

Department  of  the  Army 
Automated  Logistic  Management 
Systems  Agencies 
St.  Louis,  MI 

Department  of  the  Army 

Naval  Training  Equipment  Center 

DRCPM-TND-PA 

Orlando,  FL 

US  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
Alexandria,  VA 

US  NAVY 

Naval  Air  Systems  Command 
Rel iabil ity/Maintainabil ity 
Washington,  DC 

Naval  Research  Code  #474 
Arlington,  VA 

Naval  Research  Laboratory  * 

Code  7906  &  7503 
Washington,  DC 

Naval  Surface  Weapons  Center  * 

Code  K70 
and 

Dahlgren  Laboratory 
Dahlgren,  VA 

Naval  Sea  Systems  Command 
Code  63  E  11 
Washington,  DC 

*  Multiple  Technical  Inquiries 


Naval  Systems  Weapon  Command  * 

Code  K74 
Dahl  green,  VA 

Naval  Training  Equipment  Center  * 
Code  N-4133 
Orlando,  FL 

Naval  Underwater  Systems  Center  * 
Codes  3545,  3552,  411,  715 
Newport,  RI 

Commander  Frank  Boebert,  USN 

DCASR 

NY  NY 

OTHER  GOVERNMENT  AGENCIES 

Federal  Aviation  Administration 
Margate,  NJ 

General  Services  Administration  (GSA) 
Automated  Data  &  Telecommunication 
Service 

Washington,  DC 

Department  of  Heal th .Education ,  & 

Wei  fare 

Public  Health  Service 
Bethesda,  MD 

NASA  -  AMES  Research  Center  * 
Moffett  Field 
CA 

NASA,  Goddard  Space  Flight  Center  * 
Greenbelt.MD 

National  Bureau  of  Standards  (NBS ) * 
Washington,  DC 

National  Security  Agency  (NSA)  * 
Ft. George  G.  Meade 
MD 

Nuclear  Regulatory  Commission 
Washington,  DC 
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Elementary  Particle  fnformation  Ctr 
Oak  Ridge  National  Laboratoy 
Oak  Ridge,  TN 

US  General  Accounting  Office  * 
(several  locations) 

US  Environmental  Protection  Agency 
Research  Triangle  Park 
NC 

US  Department  of  Transportation 
Office  of  Automated  Systems  Policy 
Washington,  DC 

US  Department  of  State 
Fairfax,  VA 

UNIVERSITIES 

State  University  of  New  York  at 
Binghamton 
Binghamton,  NY 

Case  Western  Reserve  University 
Cleveland,  OH 

Colorado  State  University 
Ft.  Collins,  CO 

Florida  Atlantic  University 
Boca  Raton,  FL 

Georgia  Institute  of  Technology 
Atlanta,  GA 

The  John  Hopkins  University 
Laurel ,  MD 

New  Mexico  Technology 
Socorro,  NM 

University  of  Rochester 
Rochester,  NY 

University  of  Missouri  * 

Rolla,  MI 

*  Multiple  Technical  Inquiries 


University  of  Minnesota,  MISRC 
Minneapolis,  MN 

Syracuse  University 
Syracuse,  NY 

FOREIGN 


Bell  Northern  Software  Research  (BNSR) 
Toronto,  Ontario  Canada  * 

The  Royal  Bank  of  Canada 
Montreal,  Quebec,  Canada 

The  Toronto-Dominion  Bank 
Toronto,  Ontario  Canada 

Standard  Bank  of  South  Africa, Ltd 
South  Africa 

CCSL 

Systems  Engineering  Controller 
Austral ia 

Depto  Metodoc  Quantitativos 
Brasil 

GMD  Institute  for  Software 

Technol ogy 

Germany 

Hindustan  Computers  Limited 
Bombay 

Japan  Software  Industry  Association 
Tokyo 

El  rand  Information  Systems  Ltd. 
Jerusalem 

City  of  Toronto  Computer  Service  * 
Toronto,  Canada 

Vrije  Universiteit 
The  Netherlands 
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Universidad  Nacional  del  Su r 
Argentina 

Universite  Paul  Sabatier 
France 

Universita  Degl  i  Studi  DI  Milano 
Milano,  Italy 

Fourth  Transportation  Brigade 
Camp  King,  W. Germany 

New  South  Wales  Institute  of  Tech 
Austral ia 

Mr.  Tom  Gilb,  Consultant 
Norway 

Marconi  Avionics  Limited 
London 

Ministry  of  State  Urban  Affairs 
Ottawa,  Ontario  Canada 

Nippon  Electric  Company  Ltd. 

Japan 

Runit 

Computing  Centre  at  the  University  of 
Trondheim,  Norway 

John  Swire  &  Sons  (U.K.)  Ltd. 
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Statistics  on  Keyword  Distribution 


This  Appendix  contains  an  alphabetical  listing  of  the  475  keywords  in  the 
DACS  Thesaurus  together  with  the  number  of  documents  assigned  each  keyword. 
There  were  8164  keyword  assignments  made  to  1346  online  document  descriptions, 
yielding  an  average  of  six  keywords  per  document. 

The  keywords  assigned  most  often  are: 

Keyword  Number  of  Instances 


Support  Software 

121 

Structured  Programming 

no 

Reliability 

103 

Developmental  Tools  and  Techniques 

101 

Development  Management 

98 

Testing 

91 

Automated  Verification  Tools 

89 

Specifications 

85 

Program  Correctness 

83 

Configuration  Management 

81 

I 


Q 

d 

O  CO 

nr 

*— 4 

CO 

rH 

•—i  LO 

LO 

f-i  CO 

cvj 

LO 

HOONCC 

7 
1 

3 
1 
5 

8 

4 
9 

CTl 

m  in 

nT  CM 

O'* 

cr» 

UJ  QL 

X  o 

CM  CM 

nr 

iH 

iH 

r-4 

rH 

r—4  f— 4 

r-4  CM  nr  CO 

CO 

CM 

CM 

LU  ji 

d  >- 

2: 

LU 

►—4 

22 

OO 

OO 

►— 

1— • 

CL 

2: 

O 

UJ 

h~ 

CL 

s: 

CD 

W 

CD 

LO 

0 

2: 

2: 

0 

►— < 

O 

Z 

0  00 

*—4 

LO 

LO 

c 

d 

h- 

2: 

22 

-J 

• 

«=C 

LU 

CL 

CL 

0 

O 

J — 

O 

2: 

• — « 

CO 

h- 

— J 

>- 

K- 

z 

Z 

LU 

CL 

d 

LO 

LU 

2l 

O 

00 

< 

CD 

CL 

* — 1 

2: 

21 

LO 

O 

H 

LU 

— t 

>- 

H» 

LO  Z 

_J 

CD 

CJ  CD 

00 

U_ 

< 

h-  5: 

21 

LU  O 

LU 

z 

uu  2: 

Z 

LU 

►*~4 

♦—  <c 

CU 

LU 

CL  M 

> 

— 1  >—i 

to 

O 

Q 

CD 

0 

LO 

—1  LU 

K— 

2: 

ZD  h- 

LU 

—1  2: 

_ 1 

*— 1 

CD 

21 

2: 

LU 

C  h- 

•-4  LO 

LU 

f—  C 

d 

f 

0  2: 

LU 

h- 

LU 

2: 

4— « 

<£ 

2: 

d 

Z  LO  S  LO  CL 

01 

CD  CD 

_ 1 

0  < 

d 

< 

—I 

* — < 

2:  0 

2 

*—4 

O'  cu 

O  LU  LO  LO  UJ 

ZD 

ZD  »—* 

2: 

cr 

CL 

oc 

O 

O 

* — • 

OO  _J 

cd  cd 

4—4 

d 

LU  LU 

*-<  O  LU  -J 

LO 

Ol  2: 

< 

CD  CD 

<c  0 

51 

*— < 

OO 

~U  _l 

* — 4 

Ll 

O 

2  21 

h-  <C  l/-)  z  4-, 

c 

h-  d 

CL 

LO  O 

1—  0 

_ 1 

OO 

LU  LU 

OO  r— 

<  z 

*—  2; 

<D2LJQ. 

LU 

lo  s: 

CD 

<C  CL 

«=:  oc 

>- 

»• 

C- 

4— » 

d  d 

LU 

d 

2  O 

< 

CD  CD  O  O  >  21 

5C 

s: 

O 

CL 

CL 

O  CL 

h- 

»— 

CL 

2. 

O  O  Q  O 

2 

1 — < 

LO  CL 

Jr  M  2  M  M  O 

21  0 

CL  LO 

*~« 

*— • 

c 

21  Z 

2: 

<c 

>-  h- 

-  CD 

t-H  L.  <  h-  h-  U  >- 

>- 

O  CD 

CL  LO 

h- 

»— 

O  O 

— 1 

O 

CL 

4—4 

4-  < 

>  O 

QH  J<<  1  LO 

1— 

»— 1 

LU 

Z 

Z 

•— 1  *— « 

* — » 

*— * 

00 

►—4 

21  QC 

ZD 

O 

LO 

4~ • 

O 

LU  CL 

<  CL  CD  CD  CL  a:  *— • 

►— « 

f —  a: 

CL  Z 

LU 

LU 

CJ 

cu 

CO 

00 

J — 

«£  O 

LU 

LO 

— J 

» — 4 

Z=  CL  >- 

ULJQmwUUX 

X 

<  LU 

LU  LU 

oc 

CL 

oc 

<r  <  <  c 

4—4 

OO 

>—*  > — • 

21 

LU 

OO 

LU 

4 - 4 

Ll 

LO  J— 

o:>22zuuijj 

LU 

h-  K- 

h—  LO 

CL 

Ol 

0 

2l  Z 

-j 

_J 

z 

>—< 

O 

OO  > 

O 

00  a  2: 

00 

4—4 

X  L-  — <  _J 

<DD*-«mu 

— J 

d  d 

d  >—• 

d  d 

O  O 

►— « 

4— « 

0 

_ 1 

4— 1 

U<h 

_ 1 

►— « 

C  h- 

COLUCLOLULU2:5-2:a.CLQ. 

CL 

CL  CL 

CL  CD 

CD  0 

>- 

»—  4-  < 

c 

4—1 

_ 1 

OO 

>-  m  I —  0 

♦— 1 • 

I/O 

Cl 

CL 

Lui-.<dQdss2:5:s:s. 

>z 

z  i: 

s:  z 

z 

Z 

LU 

d  x^ 

> 

> 

<  < 

c  LU  0 

ZD 

'  . 

ZD 

<  LU  X  2  _ IOOOOCOOOCJ 

0 

0  0 

0  0 

0  0 

22 

<<<< 

c 

CO 

CO 

co  00 

CO  CO 

CO 

CO 

cu  0 

CU  CD  CD  CD 

CD  CD  CD  O  CD  CD  CD  CD 

CD 

CD  CD 

CD  CD 

CD  CD 

Q  O 

r^C74COP^LOnriOCOCJiO 

r**'  CM  r-4 

rH  M  fs  O  CVJ  ^  CVJ 

CM 

»—4  CsJ 

6 

3 

2 

*-i  r-t  00  in  m  0  <r> 

LU  CL 

*— 4  LO 

f— 1 

f—i  1— «4  f— 4  »— 4  r— 4 

CM 

MiHCViHlOCO 

X  O 

LU  S 

O  >- 

Z  LU 

»—♦  22 

LO  LO 

b-  *-» 

Z  d 

LU  h- 

£ 

d  0 

CD  Z 

2: 

O 

LU 

O  LO 

►— 

d 

LO 

• 

2>- 

O 

LO 

LO 

z 

LU 

U 

CD 

LO 

O 

LU 

C 

-U 

CL 

S5 

d 

LO  O 

LO  »— 

c 

CD 

V 

Z  4— '  O 

2l  Z 

d 

Z 

1 — 

O  LO 

LO  O 

O 

<C  LU 

LO 

Z  ■— f 

•— •  h-  CU 

z 

LU  _J  LO  CD 

LU 

LO  O  ►—  _i  Z 

z  z 

c 

OC  LU  Z 

LO 

h— 

-J  •— 4  CD  C  O 

<  LU  0 

_ 1 

CO  *—4  LU 

1— 

O  h  111  Z  i—4 

x  2:  z 

3  LU  _J  CD 

z 

h- 

O  C  h-  C  h- 

CD  CD  LU  <C 

LO  CD 

J—  J —  —1  ^ 

LU 

4—4 

L)  1 —  h—  LU  <C 

LO  Z  LU  LO  CD 

•— 1  z 

Ll-  Z  2!  -J 

LU 

z 

z 

*— 1  Z  Q  2£  CD  CD 

LJm  Z  1—  <C  CD  CD 

LO  4—4 

O  LU  <  _l 

0 

LU 

d 

h—  Z  LU  C  Z  ♦“* 

Zh  LO  Z  Z  Z 

>•  i. 

LO  t— 4  Ll.  LU 

< 

J— 

LU  CD  2:  QC  OC  4— 4  Ll. 

*—•  LO  J  O  <C  *— « 

-J  2: 

CL  t— 

d 

< 

0 

OC  4— 4  0  O  CD  h—  •— 4 

ZUJXOUZ  f—  z 

c  c 

♦—  O  -J  Z 

CD 

LU 

O  LO  CD  OC  O  LO  QC 

CD  f“  h-  CL  >“  LO  CL 

Z  CL 

d  1  LO  <  LU  *-h 

z 

LO 

h- 

LU  LU  O  OC  OC  LU  LU 

5C  J—  Z  Z  LU  C 

C  CD 

CD  Z  Z  OC  OC 

c 

< 

IQQUO.h> 

ZLU-UZOO*— <»—  3 

O 

♦-H  O  O  d  d  -U  LO 

-J 

LO  I— 

z 

f-- 

CD  4—4  O  ^  ►H  U 

2~  OC 

jMMhh<a 

z  z 

0 

Q  d  0  d  Q  d 

Z  CQ  CD  h—  K-  4—4  LU  LU 

X  CL 

Li_  h“  h—  CD  CD  4— 4  LU 

>- 

O  LU 

H-  <C 

<C  LU  LU  LU  UJ  LU  LU 

d 

CD  «C£  •— 4  1  •— 4  *—4  Qj  >  Z 

H-  O 

<<UUJU_I_I 

4-4  2 

d  h- 

4—4—1 —  1 —  1 —  1 —  4— 

CL 

O 

<Cf—  LOLOLOLO<Ci— iQl 
CCQ.(/)LO*-44-4(-HOU 

»—»  CL  C 

OUOhl-i-ifflg 
— 1  '  •— 1>— 1>— |U-S£ 

h-  z 

OC  CD 

f  £§£!££ 

2 

h-ULiJUDDD.  Q.CQOC  i 

— D  — J  — J  d  DZ  4—4  LU 

LU 

LU  »-4 

— *  O 

0000  0  0  0 

>- 

couuuaa<  <  a  0  oa  u 

0  Q.  Q.  U  U  K  LO 

LO 

LO  LO 

Ohh 

(-t-(-4-4-(—4- 

LU 

QJUUUUUQQm  J 

u  u  a 

CL  CL  a.  OC  OC  CL  LO 

in 

LO  LO 

d  d  d 

LC 

<<C<<<C<<C 

<c  < 

<  c  <c 

<<<<<<< 

B-2 


'(  , 


r^^rcsj,— inhMi- II— i  ^  IX  03  CO  I— (Oil— ihshOOCCOh 

ro  oo  «j-  hnhi —  i— i  uo  cv  vo  vo  o  •— t  io  n  h 


0  2 

CO 

h- 

0 

LU 

CD  CO 

ZD 

h- 

13 

O' 

2 

• 

2 

t— « 

UJ 

LU 

O 

0 

2  2 

CD 

z: 

Z 

►— < 

c  m 

< 

LU 

CO  h- 

.  CD 

ZD 

_ 1 

LU  C 

a:  lu 

CD 

d_ 

ZD  CD 

CO  <h 

co 

CO 

O' 

LU  CL 

_J  • 

< 

X 

.  U_ 

QQ  O 

0  co 

_J 

LU 

Z  — 1 

►—  CD  *— «  2 

CL  >-  CD 

H— 

CL 

ZZ  CD 

2  0  _l  C 

_J  2 

CL 

CD 

CO 

O 

CO 

CD  LU 

LU  _/ 

2  C  *— i 

LU 

2 

>- 

»— 

UJ 

lu  Cl 

2IQC0I-O0 

O  2  CO 

Q 

t-H 

CO 

• — 1 

1— * 

h-  CO 

LU  O  CO  CL  — J 

CD  C  CO 

CL 

>-  CL 

CO 

2 

CD 

LU  O  O  LU  O  O 

UJ 

O 

CD  ZD 

CL  LU 

O 

CO 

CO  CO  0 

0  cl 

D<  ZUdO 

»—  CO  CD 

CO  2  h- 

LU  CD 

51 

CO  CL  LU  _ 1 

2  0 

CD2K-OO.J— 

u-  2:  0 

x 

C  O 

h-  c 

LlJ 

*— «  UJ  0  O 

CO  C  •— • 

>-  C  lu  a:  zd 

<C  lu  a: 

CD 

CO  CD  ZD 

ZD  ZD 

U_ 

w  n  <  □  3  >  u  n  s;  CL  U1  _l  C£  h(LZ 

>>-DOujM<  C  (_>  </0  O 

CO-Q- 

<<Z(—  >OLlJZZZZZZZ>-'— iWUH 
ZZ<UUJCmuUJUILLlUJUJUJh<  t—  Cl 


□  o.a.a.a.fi.aa.3_jhCQz 
zzzzzzzujooooooocjcuji-illj' 
ooot3c3C;c:a:_i_i_j_i_j_j_:— it— acoes; 
►— «  » — <  »— I  t— «  »— •  ►— «  ►— <  •— «  LU  LU  LU  Lu  LlJ  LU  LlJ  U_  •  CD  h—  Z5 

l.l  111  LtJ  L.J  LJ  UJ  UJ  111  IiJ  LlJ  UJ  hi  LlJ  1.1  UJ  >H  »— >  *— *  O  ' 

QOOaoaCQCieQCQOOOCJ'^OQClQOi 


H  Z  21  H  t  21  -I 

Q_  Q_  Q_  Q_  0_  Q_  QQ 
O  O  O  O  O  O  CJ 


>-  z:  qc  q_  o  1 

—1  x>  t—  51  z 

C  O  00  o  c 

ZkiUJ  >-  t_i  I  I 

c  oc  oc  z  o  z  ; 

OZGQO 

UUOi-iUJLUUJih 

— •!—  *-iOOt—  1 
SE 22<OQQ< 
CCCCO— IUJUJ_J| 
LliZZZDLlILCjZ 

qcs->->-ciu-s:2:s;' 

QQCQUtiJUlLJLU 


CM.~iCMOCLnvouovOi— ivooaQi— i«3-Lncocr^cocMn.— ijocor~-vGCMir>CGro«j-i-ir'Ovooc; 

00  CO  — I  VO  CNJ  LO  CM  C  •— I  Cl— I  COC  i-iCMCOCO  — 1  LD  C  CM 


10 

<✓0  1 —  t— 
O  C/0  1/0 

•-<00 

coo 


<✓0  t/0  C/0 

accLti: 


z  z  z 

O  O  O  1 

000 


o  =>  z  z  o-  z 

cc  QUO  •— 1  c 

a.  LlJ  >  1— 1  z 

X  1-1  h  CO  Ih 

C/0  O  t—  C  OC  LO  O  ■— I 

1/0  (/OOSlO-JLuLu 

LlJ  LlJ  •— 1  t—  UJ  I—  LlJ 

Z  O  Lu  t—  O  O  Z 
t—  ZLuLOCOOLlJ 

o  clliull2:zc: 

LlJ  •— *  I 

Zt—  t—  h-h-t—  t—  t—  t— 
OCLOLOLOLOOOLOLOLO 
OOOOOOOOO 
OOOOOOOOO 


Z  Z  LO  C/0  CO 

00  z  a.  z 

1/0  •— 1  >— 1  Q.  LU  I— > 

Ht-l-lflCt-V) 
C/0  O  O  Z  OC  LlJ  00 
>.UIUmO£LJ 
C  _l_l_ICZCO 

—I  C  _ I  — i£Oko 

o  zooo—ica; 

O  COOOU.Q.Q. 

a-  cccccccc 

CC  LO  t—  I —  h—  t —  t—  t—  h—  h- 

OCCCCCCCCC 

OOOQOOOOOO 


c  i- 

*  Z  O  LlJ 
O  •— I  CO 

—  —I  c 

1  I—  Q-  Z 
C  CL  c 
o  c  s: 


Z  QC  C/0  c/0 

c  c  c  c 


«t  C/0  C/0 

1  =»  c  c 

0Q  CO 

c  c  c 

h~  h—  h~ 

c  c  c 

000 


KEYWORD  NO.  DOCUMENTS  INDEXED  KEYWORD  NO.  DOCUMENTS  INDEXED 

USING  THIS  KEYWORD  USING  THIS  KEYWORD 


'31VDO^r-(C\|^-(^in^lOCMC\JiH«a,NOO^^OOVDri^rHCOCOlOrHrHlHfV.^»HrH 

i-H  CO  CO  i-H  •— I  10'  CNJ  CO  *-H  «-•  CVJ 


O 

I 

oo 

CO 


LU 

CU 

00 

o 

z 

cl 

o 

z 

a. 

00 

»— * 

o 

i 

z 

h* 

►— « 

h- 

o 

OO  <£ 

i — 

X 

i — i 

o 

o 

CL 

i — 

LU  *— < 

LU  LU 

2 : 

x 

z  -u 

CL  h- 

o 

» - 4 

CQ 

h-  a- 

c  o 

2: 

LU 

*— < 

O  CL 

3:  cl 

•— < 

»-  a; 

oo  oo 

CL 

LU  OO  C 

H  Q_ 

t—  =5 

LU  X 

h-  CD 

CL  Z 

Ll. 

2L 

O  —1 

OO  Z 

CL  O  OO 

o  z  z 

< 

— u  cu 

<C  Cl 

CD  •— * 

O  OO 

oo  o  c 

o  cl 

*— •  X  X 

ZO 

ZQO 

UhlL 

LU  21  UJ  i— <  •— I 

21  o 

oizua 

o  >- 

CD 

CL  CD 

O  O  O  h-  H- 

*—  o 

S  <  O  (X  1— 

Z  X 

CL  O  X  Z  Z  LU  O 

2kh2Z<< 

cl 

<  oo 

<  o 

LU  ^  CQ  O  O  00  QC 

<I-<<UU 

OO  Q- 

-U  — J  J—  LU 

—1  CL 

LU  CL  LU 

OO  Cl 

C  u  a  o. -"-i  v 

LU 

LU  O  00  — 1 

< 

Z  h-  O  H:  h-  c 

LU  LU  LU  U_  U-  h-  h—  _J  OO  QC  Z  <  __J  C£L  — ■  LU  <  <  _/ 

O  j—  _l  —J  cC  OO  -  CU  —i  >-  LU  LU  o  51  h  h  h-  liJ  < 

>  LU  O  O  Q  O  — J  Z  Z  Z  DuU.mii>m  Z  O  O  Z  Z  >  •— • 

<COf —  ►—  00>— »ZOOO  •<  CL  h  U  U  LU  X  LU  LU  LU  LU  LU  CL 

»  z  5:  co  <►-"-•  «  u<oo)aa  j  *w-h-  o  li_  5:  s: »—  h- 

» —  I —  I —  » —  ^ah(-h>^h3aH<<  lj  z  a;  o:  u  lj  o  (/) 

JJJUUUXhUUUlONOOQ<CCQ:Q:IOQ<UUJUDD 

DD3DUUUJaZZZQ.UUaN3liJlUOQ _ iSQ-O-O-Q-CQ 

<<<<>hmJ0DDD>“<<<<UJmm(-,h40D>25:S:2Z 
LuLi_u.Luu_u.Luu_u.Li_Li_03i^:3i3:2:2:ni^:3r3:^:3i»“.— •  —  — 


Cn^f^lSrHNi£M£)Na^^C^WCOONOOYfOC\U/)rHinrOrHCOC\jOO^- 
rocvjcvj  hh  csj  his  ^  .-h  *-i 


CO 

LU 


FAILURE  RATIO  16  INFERENCE  RULES 

FAILURES  14  INFORMATION  HIDING 

FAIRNESS  2  INFORMATION  SYSTEMS 

FAST  1  INPUT-DATA  SENSITIVITY 

FAULT  6  INTERFACE  CONTROL 


U5i— i  oo  cm  cm  co  — if— ir i  ix  oj  fo  k  «  i— uncoai<r^,'iMMo,v)w^,'CCv|fl,r'lfl 

— i  cm  ^  ih  (\  s  ro  v  — t  co  cm  cm  co  — i  — i  cm  ■— 1 


o 

00  . 

o 

UJ  ' 

e: 

h— 

0 

z 

►— ( 

*—4 

OO 

y 

►— 

00 

on  _i 

O 

0  00 

00  >- 

-J  LU 

*— 4 

<  LU 

Lu  CX 

LU  Q 

h~ 

a:  ex 

z 

CD  O 

o  o 

c 

CU  ZD 

0 

O  UJ 

O  51 

-J 

O 

•-H  O 

CL  ZL. 

ZD 

cd  uj 

z 

a.  ►— 

► —  >- 

51 

Z  O 

*— H  ►— « 

Z  QL  H- 

•— *  O 

00  2T 

c; 

Z  LU  LU  LU 

UJ  LU  OO  ►—« 

00  oo 

S  a 

O  £ 

00 

z 

OOOO 

2:  Lu  CX  U 

ex 

£  Q- 

0-  c 

Z  O 

*— * 

•— «  <C  <C 

LU  on  LU  *—4 

o  oo  oo  o 

c  >- 

Z  CL 

O  00  CD  Z 

2l 

00  =D  =D  Z=> 

CD  2  Ui  H  CL 

oo  2:  o:  z 

CX  h-  Z  Z 

0  0 

*-»  >-  z  ~ 

£  — 1 

ZD  CD  CD  O 

c  C  o  =D  C 

00  <C  LU  c 

CD  — •  O  O 

CD  O 

►—  __J  »— «  h— 

<C  LU 

-J  z  z  z 

za:<co.— 

Ljah- 

C  — J  *— < 

LU  CC  >~ 

C  <  N  00 

a:  0 

u<<< 

<  h-  ID  S  -J  u 

UODO 

a:  h- 

oahNz 

c:  0 

X  J  J  J 

5!  13OUOOO0.Z  Q-Cfl<<  «M<Wh  l/~. 

iu2uaoaci;i;«  coo  cc.  cc  cc  cc  oo  ex 

>-OC  uaaOJi/>z>-<MM<<<<<uuijjujo 

CtC<C-JOOOOOO_J  _ l£XLl_U_U _ l_J_l_J_J_J_l_l_J(— 

OtOCOCCCCidCa:*— *LULiJUJ>— II— II— i3333333Dw>i 
£O0(—  OOOOOZGOOOQCSCiCiCXaaCJOOCSZ 
UJUJUJ—i— OOOOOOCQQOOQOOOQ 
£s:£££:s:3Li:£:££:i.£;s;z.££££:2:££i:££ 


Wh-  </0  C  £  UJ 

IflK  I/IOC  _J  _J  — J 

uj  lu  ui  o  O  C_  OO  — J  <C  <C  C 

_i  _i  _i  h-  — ii-i-  c  oc  ac  cc 

i  »  •.  ►— <  I —  I —  C  I —  XJ  X>  XJ  XD 

QQQZ  — J  _ /(/)</)! —  I —  I —  > — 

o  o  o  o  =>  =>  =  =  =>  <  «x  «x 

£  ££HS;£ZZZ 


rtH(\|Hn(M«'OHUl<tBCIflO®CMCOlftN(,)Mn<TCOOa)C)M^<CV'-i' 
» — I  .-I  M  ID  CM  C  CM  «3"  CM  — i  CO  UO  CO 


O  b- 

o  z 

Q£  < 

O.  «-i 

CCQCQ 

U  <  <  UI 

t-  >  cc  > 

Z  Z  CO  c 

i  — »  t~*  ►— « 


os:  t- 

o  £  oo 

z;  o  >- 

O  00 

< 

a  oo  a 

z  o  UJ 

C  — '  00 

CC  I—  < 

O  oo  cc 

£  •— 

I  O  UJ 

• O  ts 
S2  — I  o 

00  _J  UJ 
Z  t—  <  — I 
I— I  Z  — i  3 
J«>0 

UJ  o  o  z 

on  oi  sj: 


<  00  00 
O  t—  _J 

I— .  00  >-  00  o 

JKl—  O  O 
a-  O  I—  Uh 

a.  oo  _i 

<  (/)  M  UI  UJ  UJ 
UJCQOOOt— 

OO  o  c  z  z  z  z 

O  O  Z  C  <  «C  UJ 
1—1  OC  — i  Z  Z  Z  £ 
I- a.  <  uj  uj  uj  uj 
IflOl— hhh-O 
— 1  CC  z  z  z  z  c 
,  C  U  ^  M  •— «  z 


O  — <  Cj 

O  oo  _J  UJ 

►—  _i  uj  oo  o 

UJ  V  I- 

t—  O  — 1  Z  -J 
20  JUI< 
UJ  £  £  O 

£  sew  — 

UJ  >  D  cc  z 

o  o  £  =5  c 

<iC— MX 
Z  CC  X  C  UJ 

£££££ 


KEYWORD  NO.  DOCUMENTS  INDEXED  KEYWORD  NO.  DOCUMENTS  INDEXED 

USING  THIS  KEYWORD  USING  THIS  KEYWORD 


C*HfOKir)^^l^'£)CO»Hf^rHC^l£)»-»oOOOVOOOrsCCf^COrxCH\Jr^CSIf^COCOO'UT 
Csj  CO  •— «  nH  CSJ  M  in  ri  •— «  <M  CO  >sD  f-H  «H  »H  CO  VO  CM 


—I  00 
Cl  LU 
Cl.  —< 
— "CD 

o 


lO 

>- 


UJ  u 

oo 

o') 

>- 

UJ 

o 

> 

co  o 

2: 

z 

CD 

CD 

00  Qd 

H- 

c  o 

LU 

o 

CD  *— 

♦— 4 

LU  ZD 

z 

z 

z>  o 

h- 

ZL 

►—4 

z  > 

»— 

LU 

LU 

ID  OO 

UJ 

o 

Z 

GO 

co  2; 

GO 

o 

h — 

•— *  ►—4 

z 

CD 

CD 

Cr 

2: 

*— 1 

>-  o 

oo 

z  *— 

>- 

z 

< 

Q  Z  h- 

LU 

«£ 

<C. 

•  H- 

LU 

1— 

LU 

z 

oo 

K- 

O  oo 

2: 

Z  O  CO 

Qd 

DO 

ZD 

z  z 

Qd 

LU 

ZD 

*— •  H- 

z 

o 

oo 

-sc 

►— «  »— 1 

Qd 

CC 

00 

CO 

CD 

3C  LU 

ZD 

CD 

CO 

X  cd 

h* 

*— H 

LU 

>- 

o 

► —  00 

o 

o 

h*  h*  O 

cc 

o 

z 

Z 

CD  2: 

00 

Z 

OO 

00 

*-H 

LlJ  <C 

o 

z  z  z 

oo 

*— * 

g 

CO  LU 

z 

Lc 

00^0 

c 

>— H 

< 

c 

LU  UJ 

< 

< 

Qd 

o 

Qd 

— 1  O' 

UJ 

CO  co  o 

z 

_ 1 

Li_ 

lu  m 

►— 4 

00 

CC  D  CC 

< 

u 

H-  CD 

LU 

Qd 

O 

l-H 

J— 

Cl.  h- 

OC 

►— *  »— 4  *— » 

LU 

cc 

►—4 

h-  h- 

h— 

z 

LU  *— 1  Q_ 

oo 

c 

2: 

Z5 

h- 

Qd 

00 

s:  z 

cc 

CO  oo  oo 

*— 

2 

cc 

Q 

o  z 

00 

< 

O  —I 

— 

CO 

CD 

CO 

CD 

CO  z 

OO 

CD 

h- 

»>— 4 

o  o 

o 

LU  UJ  UJ 

X 

< 

»— H 

o 

cc  >- 

LU 

Qd 

Z  c  Qd 

cc 

z 

z 

2d 

z 

2  £ 

z 

z 

00 

< 

LU 

o 

O'  o 

cd 

d  O  Q 

LU 

u_ 

_ I 

z: 

a_  oo 

h- 

H- 

r>  >  lu 

LU 

►—4 

►-H 

H-H 

►— « 

o 

o 

oo 

Lu 

2E 

2.  2. 

2 

£££ 

2: 

2 

2 

2. 

2  2 

2 

5: 

2:^1 

1 

£ 

i 

2: 

2 

*— 4 

1— 

►H 

-D 

o 

>- 

>- 

>- 

Z 

>-  O  CD 

<  < 

<c 

<<< 

c 

c 

< 

<c 

< 

<c 

<  c  «c 

<C 

c 

< 

< 

< 

<<U 

CD 

CD 

CD 

»— 

»— 

h- 

1—  z 

»— t 

Qd  CC 

Qd 

Qd  CC  cc 

Cd 

Qd 

Qd 

O' 

Qd  Qd 

Qd 

Qd 

Qd  Qd  Qd 

Qd 

Qd 

Qd 

Qd 

od 

Qd  Qd  LU 

LU 

UJ 

O 

►H 

i 

*—h 

»— i 

•— 1 

LU 

co  co 

CO 

O  CO  CO 

CO 

CO 

8 

tn 

CO  CD 

CO 

o 

CO  CD  CO 

CD 

CO 

o 

CD 

CO 

CO  CO  'D 

h- 

t— 

H- 

u 

_ 1 

-J 

_ 1 

ZD 

—1 

o  o 

o 

o  o  o 

o 

o 

o 

o  o 

o 

o 

o  o  o 

o 

o 

o 

c 

o 

o  o  o 

O 

O 

O 

«c  < 

C 

<u> 

cc  cc 

cc 

CC  OC  CC 

Od 

Qd 

cc 

Cd 

Qd  Qd 

Qd 

Qd 

CC  Qd  Qd 

cc 

Qd 

Qd 

cd 

Qd 

Qd  Qd  Qd 

od 

Qd 

Qd 

ZD 

ZD 

Z> 

=> 

=> 

< 

CL  CC 

CC 

aCLQ. 

o. 

a. 

cc 

Cl. 

a_  cc 

Q_ 

Cc 

CC  Q.  Cl. 

cc 

cc 

CC 

cc 

cc 

CC  CC  cc 

cc 

CC 

cc 

o-aaaoo: 

OCSJH^H^NVOONHfOHlOVOOCvJlONOrOrsH^CVJ^HrH 
CM  CNJ  CO  CO  CO  r-»  o  CO  i-H  no  f— ♦ 


HOOfOM-H 

cm  ro 


Z  00 
O  Z 

— •  o 


o 

Cl. 


CD 

< 

z 

>- 

00 

LU 

•-H 

CD 

CD 

#— 

z 

*— H 

CD 

f— 

LU 

CO 

K-H 

o 

to 

< 

< 

HH 

z 

u 

00 

— 1 

I— —4 

X 

z 

ZD 

ID 

oo 

CD 

•— » 

cc 

LU 

*— < 

h- 

u 

o 

CO 

— 1 

Qd 

LU 

a. 

Q 

QD 

<C 

< 

1—4 

z 

C 

o 

z 

CC 

< 

oo 

< 

ZD 

oo 

z 

H 

< 

>- 

> 

< 

h“ 

o 

OO 

< 

21 

2: 

oo 

—1 

LU 

< 

CD 

-J 

h~ 

LU 

H- 

CD 

to 

Qd 

od 

LU 

LU 

_l 

z 

C 

oo 

ZD 

M 

< 

< 

—  1“ 

_1 

CO 

o 

h- 

♦— 

UJ 

o 

> 

00 

1 — 

Qd 

z 

00 

— J 

>- 

>- 

Q 

Lu 

00 

c 

o 

00 

00 

Qd 

►— « 

LU 

LU 

Qd 

H 

CO 

LU 

h- 

h- 

>-  h- 

Qd 

Qd 

CD 

>- 

>- 

z 

CO 

00 

CD 

o 

00 

Z> 

CC 

LU 

UJ  >- 

>- 

>- 

u  o 

ZD 

a. 

< 

oo 

00 

u 

o 

z 

oo 

LU 

UJ 

O 

>- 

cc 

z 

oo 

LU 

< 

Lu 

Lu 

h- 

1— 

h- 

<  z 

O 

LU 

<JC 

M 

LU 

CD 

CD 

oo 

Qd 

►— 

UJ 

oo 

o 

UJ 

Z> 

M 

< 

4-^ 

^-4 

z  z 

LU 

z 

Qd 

CO 

CO 

z 

1— 

z 

Qd 

Z 

z 

h~ 

a_ 

LU 

Qd 

UJ 

CD 

Q 

o-_i 

oo 

O0  > 

> 

> 

c  < 

OO  CD 

o 

z 

z 

o 

5 

o 

CC 

£ 

§ 

LU 

oo 

-J 

CO 

Qd 

l-H 

4—4 

2  2 

Cd  o 

*—« 

Qd 

M 

M 

*-H 

X 

Z 

Qd 

z 

1 

LU 

2 

ZD 

to  oo 

to 

00  H- 

h“ 

i- 

1- 

1— 

H 

o :  Qd 

00 

C 

>“ 

►- 

p 

»— 

u 

UJ 

Qd 

Qd 

UJ 

o 

cc 

-U 

UJ 

Q 

OO  to 

oo 

00  CD 

CD 

CD  O  CD 

CD 

C  C 

O  Q- 

Qd 

UJ 

< 

c 

< 

2 

M 

< 

O 

O 

t— H 

h- 

oo 

c 

»— 4 

— 1 

UJ 

UJ  UJ 

UJ 

UJ 

ZD 

ZD 

Z3  ZD  ZD 

ZD 

Qd  Qd 

3C  l 

LU 

_i  -J  Qd 

Qd 

Qd 

l-H 

►— 

O  X 

Lu 

Lu 

Qd 

►—4 

z 

oo 

h- 

> 

cc 

CD 

CD  CD 

CD 

o  o 

O 

QQQ 

O 

CO  CO 

1—  Z 

D* 

CD  <  LU 

LU 

LU 

h- 

g 

00  1- 

od 

Qd 

h- 

1—4 

♦—4 

Qd 

l—H 

o 

O 

O  O 

o 

o  o 

o 

o  o  o 

o 

O  O 

LU  O 

1 

ZD  CC 

CC 

CC 

cc 

CC 

c  < 

LU 

LU 

u  JO 

o 

O 

Qd 

Qd 

Qd 

Od  Qd 

Qd 

Qd  Qd 

Qd 

Od  Qd 

Qd 

Of 

Qd  Qd 

z  z 

Z 

zoo 

o 

o 

o 

cc 

a.  a. 

CC 

cc 

Q_ 

cc 

CC 

cc 

Cl. 

cc 

a. 

CC 

CC  CC 

CC 

CC  CC 

CC 

a. 

CC 

cc 

CL 

CL  CC 

B-6 


KEYWORD  NO.  DOCUMENTS  INDEXED  KEYWORD  NO.  DOCUMENTS  INDEXED 

USING  THIS  KEYWORD  USING  THIS  KEYWORD 


CM  >— I  CM  Is-.  <  r- <  ■— <  CO  •"-!  n  <  MOO  (VJMlfi 


i/>  «i 

h-  flfic 

2  >•  CZ 
-ihHCCtC 
CtL  <-i  (_J  C 


M 

Q - 1  LU  Q  CO  Z 

*-H 

LU 

— •  o 

CO 

LJ^OZ  J  LU 

o 

►— 

h-  ' 

z 

=3  o  o  <  o  c  si 

z 

— '  h- 

LU 

-U  <C  CC  b—  O  h—  LU 

LU 

CO 

co  o 

co  < 

1— 

CD  L,  Q.  OO  H  <  o 

«—< 

LU  Z 

o  o 

X 

o  c 

CO 

CO 

o  c 

o_  *— 

co 

LU 

o  cd  o  cd  cd  o  z 

h- 

>- 

c 

21  Ll. 

co 

Z22ZZ2UJ< 

LU 

z 

—1 

ID  CO 

o  — 

LU 

_ 1 

— 1 

LU 

«c 

CD  — J 

CD  cx 

z 

o 

ujQiaQcocaocz 

o 

s: 

z 

z  o 

LU  LU 

CC,  WIUJLUUJLULULUUJ 

O  <LUUIUJliJLiJU» 

'  z  cozzzzzzo: 

_l  1/1  ,i— i.— ii— iLU 

UJ  t—  OCOOOOOO- 

i  o  a  tczzzzzzrx 

O  CS  C/>  Z  QUJUUJLUUJLiJUJ 

£z  a:  < 

•— «  o  v — *  UJ  LlJ  UJ  UJ  LjJ  LU  LU  LlJ  UJ  I 

-j  z  -j  I—  OLcx.cx.Qccx.ezec.cc.ct. 

z2[C_i_JZlu0333333333 

tfCZIDD^CJCGI —  I —  h—  f —  H-f—  f —  f —  ( —  I 
2.002I2EN*— •OU-U-Li_L_U»Li»L»_Li_Li_l 
i  Ul  X  »-<  *- «•—*»— »_jZOOOOOOOOO' 


UJ  Ul  < 
CC  CD 

*-«  z  ^ 

D  UJ  < 

O'*—*  uj 
LJUZ 
QD  CO  CO 


<  o  o  > 

Jh  z 

co  co  co  o 

S'  2:  Z  2  2 

O  O  O  O  O  I— 


CX  CX  CC  I 

<  c  «£  co 
3  3  3  0 
f —  / —  J —  or 
Ll_  Lu  U_  c: 
O  O  O  Q- 
(/)</)(/)(/) 


•HOO^cvjvDcocco^i^o^csjc^oo^ocvjroMcoococsjc^rHiHCTOoNCvj^-iHinrvtr) 

C\J  OrHrHCO^r- •  LfO  *3“  C\J  C\J  CVJ  PO  «— «  — « 


vo  a  a  i 

_  *-•  UJ  UJ  I_  _  _ 

►—  ►—  UJ  H-  CX  l/)UUJ<UJUJLLM 

<<a  i/)uuj  >  i  2  ^  j  j«- *  o 
£  3  3  J  »  U.  jmmOCCCDU 

h  j  V)  bj  Q  U.  CCDOZOOUJO 

h<<ouj*-i  22:z<a:ttc.z 
(/)  >  u.1  Q  a:  O  <  uj  uj  _j  q_  Q-  oo  < 

LU  LU  2E  2~  Q.  I 

I  VObObOVOVObOVObOl/) 
I—  h-  H-h-l—  H-Z 


«c  < 

CO  CD  C. 

s:  ■-«  o 


CO  *— • 

H 

*2  CO 


co  z 
z  o 

•  >-  o  *-* 
:  ac  »-*  co 
j  uj  co  co 

*  >  a:  uj 

>  o  sa 

>  o  o  o 

J  UJ  UJ  UJ 

:  oc  os  a: 


CQ  CQ  CL  CC  QQ  CC  CQ 

<<<<<<< 


h-h-H-ZZZZZZ 

*— <  *— <  *-h  UJ  UJ  UJ  UJ  UJ  UJ 

-i-j-jzzzzzaL 

►HMMUUJUUJUJLJ 

CQCcccaactaaa: 

<<<MMMMMIH 


UJ  UJ  UJ  UJ  UJ  UJ  UJ 

cc  cx  cx  cx  a:  ex  cx 


oooo 

UJ  UJ  UJ  UJ 

cx  cx  cx  cx 


ZZZZ  0>-<OC2 
LU  UJ  UJ  UJ  •-*  o  oo 
£££2:hzuj|-a 
UJUJUJUJCUJOOZ 
OCQCOCa:5>i-iOCZ30 
M  (H  KH  M  QC  —I  ID  CX  Z 
3DDDLU*-*OH> 

ooooco  co  cococo 

UJ  UJ  UJ  UJ  UJ  UJ  UJ  UJ  UJ 

cxcxcxcxcccxcxcxcx 


O  •— I  z 
<  z  o 
o  •— < 

O  CX  H- 
•-<< 
I-  51 

<;  co  *— < 
<££h 
>-  f—  UJ  UJ  co 

H-  CO  <HhUJi 

*m  t/>  o  co  co 

_j  uj  ^  >*■  >-  uj  < 

•  Z  O  C2  CO  CO  -J  , 
£flh<Z  ; 

«t  co  cc  •— «  o  i 

VOD  JK2<LJI 

uj  o  o  o  <:  ^  u> ' 
o:  qi  cc  cx  co  co  co  < 


KEYWORD  NO.  DOCUMENTS  INDEXED  KEYWORD  NO.  DOCUMENTS  INDEXED 

USING  THIS  KEYWORD  USING  THIS  KEYWORD 


CSJ  fO  H  H  C\J  I  C\J  H  N  «— H  f-H 


LU 

CD 

z 

o 

OO 

LU 

z 

Z 

OO  CD 

< 

C 

H-  C 

u 

DC 

Z  Z 

oo 

z 

CD 

LU  CD 

z 

—J 

cd 

CD 

cd 

O 

OO 

z  z 

o 

LU 

z 

z 

z 

DC 

z 

LU  C 

o 

*—« 

*— < 

o 

D. 

o 

DC  U 

h— 

o 

h- 

1 

i 

DC 

z 

LU 

oo 

O 

z 

Z 

z 

CD 

H* 

< 

z 

DC 

LU 

z  CD 

>“ 

CD 

>-  i— t 

LU 

z 

< 

< 

H- 

z 

< 

DC 

o 

z 

►— » 

crz 

o 

h- 

z 

z 

o  o 

h- 

Z 

o 

DC 

DC 

1 

cd 

CD 

LU  •— * 

l— 

CD 

LU  *-h 

o 

00 

Z  LU 

oo 

LU 

»— « 

cd 

o 

cd 

♦— ♦ 

o 

DC  fr- 

z 

cd 

z 

DC  DC 

h- 

—i 

DC 

00 

oc 

LU 

Z 

00 

o 

o 

-J 

oo 

z 

DC 

<  z 

o 

LU 

LU 

LU 

< 

z 

CD 

LU 

oo 

O  CL 

1“ 

LU 

DC 

DC 

< 

00 

c 

a. 

a. 

3  CD 

*— i 

\— 

o 

z 

Z  LU 

DC 

CD 

f— 

z 

< 

z 

>- 

z 

Ll 

o 

CL 

D_ 

3: 

LU 

z 

CL 

h-  LU 

H* 

oo 

z 

cd 

CD  Z 

CD 

< 

CD 

M 

Q 

#— < 

—l 

z  _J 

— J 

LU 

z 

cd 

>~ 

Li-  X 

s 

DC 

z 

LU 

♦— « 

*— < 

LU 

Z 

zc 

< 

o  c 

c 

DC 

o 

o 

O 

Q 

o 

1 

DC 

O  LU 

LU 

cd 

CL 

oo 

00  CD 

h- 

—I 

DC 

00 

o 

z 

LU  CD 

cd 

LU 

LU 

LU 

LU 

LU 

LU 

LU 

O 

00 

•— 1 

ru 

DC 

LU 

LU 

LU  Z 

Z 

LU 

h* 

LU 

g 

< 

DC 

LU 

DC 

DC 

DC 

DC 

DC 

DC 

Z 

OO 

cd 

z 

c 

c 

o 

O  LU 

DC 

00 

h- 

> 

H 

h* 

00 

ZD 

z 

Z 

Z 

z 

t— < 

h- 

o 

oo 

o 

cd  oo 

00 

1 

h* 

\— 

h- 

h* 

fr— 

fr-- 

h- 

DC 

> 

gd 

DC 

LU 

z 

z 

z 

z  z 

z 

Z 

z 

z 

Z 

LU 

►— « 

M  »-H 

3  O 

CD 

cd 

CD 

CD 

CD 

CD 

CD 

DC 

z 

z 

LU 

LU 

LU 

LU  LU 

LU 

LU 

LU 

LU 

LU 

h- 

h- 

h- 

fr¬ 

Q_ 

oo 

Z 

z 

Z 

Z 

Z 

z 

Z 

£ 

LU 

§:£ 

CD 

fr- 

K 

H 

fr— 

h" 

H 

h- 

h— 

<c 

< 

<  < 

ee 

LU 

LU 

DC 

DC 

DC 

DC 

DC 

DC 

DC 

CD 

O. 

z 

Z 

oo 

00 

OO  00 

OO 

OO 

oo 

OO 

oo 

h— 

>— 

H  h- 

h- 

H- 

h- 

h- 

►— 

fr— 

► — 

fr— 

h“ 

H- 

z 

z 

z  >- 

>- 

>- 

>* 

>- 

>- 

>-  >- 

>- 

>- 

>- 

>• 

>- 

00 

00 

oo  oo 

00 

oo  oo 

oo 

00 

00 

00 

OO 

OO 

00 

oo 

oo 

00  oo 

oo 

OO 

00 

oo 

oo 

oo  oo 

oo 

OO 

oo 

oo 

oo 

TACTICS  1  WEAPONS  SYSTEMS  APPLICATIONS 

TARGET  LANGUAGE  1  WE  I  BULL  DISTRIBUTION 

TELECOWUNICATIONS  APPLICATIONS  10  WELLMADE 

TERMINATION  PROOFS  7  WORKING  SETS 

ZIPF'S  LAWS 


APPENDIX  C 


THE  ROLE  OF  AN  INFORMATION  ANALYSIS  CENTER 
IN  SOFTWARE  ENGINEERING  TECHNOLOGY  TRANSFER 
A  Concept  Paper 


i 

i 

£• 

y 

M 

i 


( 


4 


/ 


THE  ROLE  OF  AN  INFORMATION  ANALYSIS  CENTER 
IN  SOFTWARE  ENGINEERING  TECHNOLOGY  TRANSFER 
A  Concept  Paper 


The  Predicament 


As  software  engineering  advances  into  its  second  decade,  the  ideas, 
principles  and  practices  conceived  in  its  first  decade  need  to  be  assimilated 
into  a  workable  set  of  tools  and  techniques  that  can  be  dispersed  to  software 
developers  for  their  use  in  the  production  of  software J  The  need  for  this 
technology  transfer  is  clear  and  immediate.  Without  the  proper  transfer  of 
software  engineering  technology  from  software  engineering  researcher  to  deve¬ 
loper,  the  software  world  will  be  unable  to  extricate  itself  from  its  present 
predicament.  This  predicament  has  been  characterized  by  Meyers  as  follows: 

"The  general  character  of  the  software  predicament  can  be  seen 
clearly,  although  consistent  numbers  with  which  to  characterize 
it  more  precisely  are  hard  to  come  by.  Because  less  expensive 
hardware  is  bringing  more  applications  within  economic  reach, 
the  amount  of  software  to  be  developed  is  increasing.  Also, 
because  more  software  is  already  in  existence,  there  is  more  to 
be  maintained.  But  the  productivity  of  programmers  is  improving 
rather  slowly,  expecially  by  the  standards  of  hardware  price/per¬ 
formance,  with  the  result  that  the  overall  cost  of  software  develop¬ 
ment  is  tending  to  increase. "2 

The  predicament  presents  a  rather  ominous  picture  to  be  sure.  Essentially, 
the  problems  of  today  in  the  software  world  are  more  difficult,  but  the  solu¬ 
tions  to  the  problems  do  not  seem  to  be  effective.  The  result  is  a  losing 
battle  if  things  continue  as  they  have.  One  reason  for  the  losing  battle  may 
be  that  the  software  world  is  trying  to  solve  today's  problems  with  yesterday's 
solution  techniques.  For  example,  several  recent  surveys  have  shown  that  the 
transfer  of  software  engineering  is  at  a  standstill  and  that  people  are  still 
developing  software  as  they  were  five  years  ago.3 

A  Possible  Solution 


"  I  firmly  believe  that  technology  transfer  is  the  primary  means  we 
have  to  combat  the  software  problems  the  industry  has  been  experiencing*"4 
Riefer's  above  statement  points  a  way  to  the  beginning  of  a  solution  to 
the  software  predicament  that  exists  today.  Technology  transfer  needs  to 
bring  new  and  effective  solutions  from  the  software  engineering  researcher  to 
the  software  developer.  Unfortunately,  the  process  is  not  quite  as  easy  as  it 
sounds.  Setting  up  direct  communication  links  between  researcher  and  developer 
will  not  insure  that  technology  is  transferred  over  these  links.  The  wholesale 
importation  of  the  latest  software  engineering  techniques,  in  fact,  just 
invites  disaster.  The  developer  needs  to  understand  the  techniques  so  they 
can  be  evaluated  within  the  context  of  the  development  environment.  This  is 
essential.  Another  way  of  stating  this  is  by  citing  Reifer's  Technology  Risk 
Principle. 
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"Technology  should  only  be  used  when  the  risk  associated  with  it  is 
acceptable  ."5 


Some  of  the  technologies  may  indeed  be  acceptable  in  terms  of  risk;  however, 
the  developer  needs  to  understand  the  technology  and  all  of  its  myriad 
applications  before  any  consideration  can  be  made  to  its  transfer  to  the  deve¬ 
lopment  environment.  This  technology  transfer  business  is  more  than  giving 
lectures  or  writing  journal  articles  about  the  latest  technologies .  It  involves 
a  certain  amount  of  information  synthesis  and  analysis  so  that  the  ultimate 
receiver  can  evaluate  the  technology's  worth.  If  no  benefit  is  perceived  by 
the  developer,  no  transfer  of  technology  is  going  to  occur.  Of  that  we  can 
be  certain. 

Some  Solutions  Mechanisms 


Although  the  technology  transfer  process  is  a  difficult  one,  there  are 
mechanisms  in  place  today  within  the  software  engineering  community  to  effect 
the  transfer  of  technology.  Wasserman,  for  example,  cites  four  major  mechanisms 
(and  their  attendant  shortcomings) .6 

1.  University  graduates  go  to  work  in  software  development  settings 

Problem:  New  graduates  generally  have  positions  of  low  visibility  and 
responsibility  and  have  not  received  any  development  experience  within 
the  university  environment. 

2.  University  faculty  serve  as  consultants  to  industry. 

Problem:  Consultants  often  opt  for  more  "interesting"  (i.e.  more  research- 
oriented)  situations  as  they  come  about  and  have  little  continuity  within 
one  company.  They  also  lack  the  same  software  development  expertise  that 
their  students  lack. 

3.  Industry  people  go  to  the  university. 

Problem:  Sabbaticals  are  not  open  to  many  people  and,  besides,  they  are 
rarely  taken  by  people  with  major  project  responsibilities. 

4.  Industry  people  attend  short  professional  development  courses. 

Problem:  The  direct  application  of  the  techniques  learned  in  these  courses 
is  often  difficult  to  transfer  to  a  typical  work  situation. 

These  four  techniques  all  involve  some  level  of  inter-personal  dealings  in  an 
environment  different  than  one  the  person  is  used  to.  Because  the  performance 
of  individuals  in  such  a  situation  has  such  a  wide  variance,  it  is  difficult 
to  obtain  a  consistent  level  of  technology  transfer.  The  crucial  process  of 
transferring  the  context  of  ideas  generated  in  a  researcher  environment  to  a 
developer  environment  is  a  difficult  achievement  when  attacked  on  a  one-to-one 
basis.  Nevertheless,  these  techniques  should  be  effective  if  the  "right" 
person  for  the  job  can  be  identified. 
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Studies  in  scientific  and  technical  information  dissemination  have 
identified  such  a  person. 1  Called  a  "gatekeeper,"  this  person  has  the  impor¬ 
tant  technology  transfer  ability  to  interface  the  outside  world  (i.e.  the 
research  community)  to  the  inner  realm  of  the  development  organization.  Much 
of  the  time,  however,  these  people  tend  to  be  senior  technical  staff  and  not 
high  level  managers  who  could  influence  the  company  to  adapt  a  new  software 
engineering  technology. 

The  professional  societies  and  the  journal  literature  serve  as  an  aid  to 
technology  transfer  in  a  more  formal  manner  than  the  inter-personal  techniques. 
For  example,  many  of  the  societies  arrange  and  conduct  tutorials  to  disseminate 
information  about  software  engineering  technologies.  Along  the  same  lines,  the 
journals  of  the  professional  societies  publish  articles  about  the  latest 
techniques.  Although  these  techniques  are  formal,  their  effectiveness  is 
limited  by  the  very  structure  of  the  professional  groups  and  journals. 

Professional  groups,  by  their  nature,  are  essentially  special  interest 
groups  created  to  advance  the  interest  and  ideas  of  a  relatively  narrow  area 
of  interest.  For  this  reason,  the  societies  and  their  journals  are  more  for 
the  purpose  of  intra-group  rather  than  inter-group  communication.  Both  re¬ 
searcher  and  developer  have  their  oun  societies,  and  both  sides  recognize  the 
need  and  importance  of  the  technology  transfer  issue.  But  the  very  structure 
of  any  society  is  not  designed  to  facilitate  a  process  such  as  technology  tran- 
fer.  The  required  interface  mechanism  between  developer  and  researcher  is  not 
strong  for  the  societies  or  the  journal  literature. 

Information  Analysis  Center  as  Technology  Transfer  Agent 

A  formal  mechanism  that  can  perform  the  interfacing  role  between  research¬ 
er  and  developer  for  the  purpose  of  technology  transfer  in  the  software  world 
does  exist.  The  mechanism  is  the  information  analysis  center.  Specifically, 
the  information  analysis  center  for  software  engineering  is  named  the  Data  and 
Analysis  Center  for  Software,  hereafter  referred  to  as  DACS. 

The  purpose  and  objective  of  an  information  analysis  center  transcend 
those  of  a  technical  information  center  or  library.  The  technical  information 
center  and  library  provide  bibliographic  services  as  their  main  commodity. 
Information  analysis  centers,  as  their  name  may  imply,  provide  information 
analysis  and  synthesis  services  as  their  primary  commodity.  It  is  these  analy¬ 
sis  and  synthesis  services  that  serve  as  the  core  of  the  technology  transfer 
mechanism  within  the  information  analysis  center.  As  has  been  previously  dis¬ 
cussed,  a  precondition  for  the  transfer  of  software  engineering  technology 
from  the  researcher  to  the  software  developer  is  a  thorough  understanding  of 
the  technology  and  its  attendant  implication  in  terms  of  risks  and  benefits 
within  the  developer's  environment.  It  is  this  essential  pre-condition  to 
the  technology  transfer  process  that  can  be  provided  by  the  synthesis  and 
analysis  of  information  within  the  information  analysis  center.  Although  the 
previously  discussed  mechanisms  are  all  useful  to  some  extent,  they  do  not 
possess  the  match  of  capability  and  task  that  exists  between  the  information 
analysis  center  and  the  technology  transfer  task.  In  software  engineering,  the 
requisite  synthesis  and  analysis  skill  are  provided  by  the  DACS  as  the  soft¬ 
ware  engineering  information  analysis  center. 
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The  Information  Analysis  Center  Concept 


The  formal  concept  of  the  information  analysis  center  was  expressed  in 
1963  by  the  nuclear  physicist  Alvin  Weinberg. ^  Weinberg  had  prepared  a  report 
at  this  time  which  was  to  serve  as  somewhat  of  a  landmark  document  in  the 
field  of  national  scientific  and  technical  information  policy.  The  information 
analysis  center  concept  did  not  originate  in  the  report;  in  fact,  the  report 
listed  over  400  organizations  in  the  nation  it  considered  as  meeting  the 
criteria  of  an  information  analysis  center. 9  Highlighting  the  contribution 
information  analysis  centers  could  have  in  managing  the  nation's  technical 
information  was  one  of  the  major  concerns  of  the  report.  An  excerpt  from  the 
report  explains  the  worth  of  the  information  analysis  center  concept. 

"The  activities  of  the  most  successful  (information  analysis)  centers  are 
an  intrinsic  part  of  science  and  technology.  The  centers  not  only  dis¬ 
seminate  and  retrieve  information,  they  create  new  information . 

In  short,  knowledgeable  scientific  interpreters  who  can  collect  relevant 
data,  review  a  field  and  distill  information  in  a  manner  that  goes  to  the 
heart  of  a  technical  situation  are  more  help  to  the  overburdened  special- 
list  than  a  mere  pile  of  relevant  documents."™ 

Although  this  excerpt  does  not  explicitly  mention  technology  transfer,  that 
process  of  "...distilling  information  in  a  manner  that  goes  to  the  heart  of  a 
technical  situation"  is  certainly  central  to  the  process  of  technology  transfer 
The  receiver  of  the  technology  must  be  able  to  understand  and  evaluate  the 
technology  on  his  own  terms.  The  analysis  and  synthesis  of  information  about 
a  technology  by  the  information  analysis  center  matches  the  need  and  is  the 
driving  force  behind  the  center's  ability  to  act  as  an  effective  technology 
transfer  agent.  The  fit  between  technology  transfer  and  the  information  anal¬ 
ysis  center  is  a  natural  one;  the  statement  Weinberg  made  about  the  value  of 
the  synthesis  and  analysis  process  is  as  valid  and  crucial  today  in  the  area 
of  software  engineering  as  it  was  in  1963  when  the  statement  was  made.  By 
synthesizing  and  analyzing  software  engineering  information  into  a  form  that 
is  comprehensible  and  relevant  to  the  software  developer,  the  DACS  has  an  im¬ 
portant  role  as  technology  transfer  agent  in  the  field  of  software  engineering. 
This  task  of  synthesizing  information  in  software  engineering  is  more  difficult 
than  it  might  be  with  other  fields  because  of  the  breadth  and  dynamic  state  of 
software  engineering  at  the  present  time.  But  these  very  characteristics  of 
software  engineering  may  result  in  bigger  payoffs  when  the  technologies  are 
transferred  than  may  be  the  case  with  a  narrower  or  more  static  discipline. 

The  task  is  harder,  but  the  potent’ =>1  benefits  resulting  from  software  engineer 
ing  technology  transfer  may  be  g re,  :r . 

A.  Two-Way  street 

Technology  transfer  is  generally  thought  of  as  a  one-way  street  from  re- 
seorc her  to  developer.  Simply,  technology,  which  originates  with  the  research¬ 
er,  is  transferred  to  the  software  developer  who  is  the  ultimate  user  of  the 
urology.  In  its  broadest  sense,  however,  technology  transfer  is  a  two-way 
•rerr.  It  is  a  two-way  street  because  the  researcher  needs  the  experience  of 
.  ;rv«»ioper  for  guidance  in  future  research  efforts.  Both  good  and  bad  ex- 

.  with  technology  provide  the  researcher  with  valuable  information  to 

•  ”  *  *  <•<  hno  logy. 


Information  analysis  centers  are  good  places  for  the  developer  to  transfer 
software  experience  information.  Just  as  the  information  analysis  center  inter 
faces  between  the  transfer  of  technology  from  researcher  to  developer,  it  can 
provide  the  interface  for  the  flow  of  experience  data  from  developer  to  re¬ 
searcher.  The  synthesis  and  analysis  skills  of  the  information  analysis  center 
are  just  as  useful  and  relevant  for  both  directions  of  information  flow.  A 
large  part  of  the  DACS  effort  is  expended  in  this  regard,  particularly  in  the 
organization  of  software  experience  data  into  databases  that  can  be  used  for 
the  evaluation  and  analysis  of  software  engineering  technologies. 

DACS  -  A  Software  Engineering  Information  Analysis  Center 

In  June  of  1975,  the  Rome  Air  Development  Center  (RADC)  contracted  with 
I  IT  Research  Institute  (IITRI)  to  design  a  center  that  would  acquire,  analyze, 
and  disseminate  information  on  software  engineering  technology.  The  Air  Force 
recognized  the  need  for  such  a  center  to  serve  the  government,  industrial,  and 
university  comunity  as  a  focal  point  for  software  development  and  experience 
data. 


DoD  has  traditionally  recognized  the  worth  of  information  analysis  centers 
Several  organizations  within  DoD  sponsor  a  number  of  centers.  For  example, 
nine  DoD  Information  Analysis  Centers  are  managed  and  funded  by  the  Defense 
Logistics  Agency,  (DLA).  In  keeping  with  the  nature  of  all  information  anal¬ 
ysis  centers,  the  centers  are  responsible  for  the  acquisition,  analysis,  evalua 
tion,  and  dissemination  of  scientific  and  technical  information  to  the  managers 
scientists,  engineers,  and  technicians  they  support.  Among  the  specialized 
areas  the  centers  deal  with  are  electronic  hardware  reliability,  metals  and 
ceramics,  and  machinabi 1 ity .  As  part  of  its  responsibilities,  each  center 
serves  as  an  information  and  technology  transfer  agent  within  its  own  area  of 
technical  expertise. 

A  contract  to  establish  the  DACS  was  awarded  to  I  IT  Research  Institute 
(IITRI)  by  RADC  in  August  1978.  When  fully  implemented  and  operational,  the 
DACS  will  provide  a  centralized  source  for  current,  readi ly-usable  data  and 
information  concerning  software  technology.  This  software  information  resource 
will : 

1.  Aid  the  program  manager  in  planning  and  monitoring  software  projects. 

2.  Supply  experience  data  to  software  research  projects- 

3.  Provide  baselines  for  software  development  methods  comparisons. 

4.  Foster  the  use  of  uniform  terminology. 

5.  Aid  in  establishing  data  collection  guidelines  and  standards- 

6.  Distill  and  disseminate  information  on  software  projects. 

The  benefits  expected  to  be  accrued  by  members  of  the  software  engineering 
community,  both  developer  and  researchers,  are: 

1.  Valuable  savings  of  scientific  and  engineering  manhours  in  locating 
data  and  information. 

2.  Rapid  application  of  the  latest  technologies  via  technology  transfer. 

3.  Elimination  or  minimization  of  duplication  of  effort. 

4.  Reduction  of  software  costs  and  improved  performance. 

5.  Minimization  of  program  delays  and  schedule  stretchouts. 
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All  of  the  objectives  of  the  OACS  are  related  to  the  process  of  technology 
transfer.  Through  the  first  objective,  the  DACS  wi 11  aid  the  technology  trans¬ 
fer  process  by  supplying  information  about  software  technologies  to  the  devel¬ 
oper  in  an  understandable  format.  The  second  objective  deals  with  the  feedback 
from  developer  to  researcher,  as  discussed  earlier,  that  is  required  for  refine¬ 
ment  of  technologies.  Objective  Number  3  will  facilitate  technology  transfer 
by  providing  a  basis  for  the  evaluation  of  technologies  in  developer-related 
terns.  Objectives  4  and  5  are  objectives  that  will  streamline  the  technology 
transfer  mechanism.  Uniform  terminology  (objective  4)  and  data  collection 
guidelines  (objective  5)  are  essential  to  aid  communication  between  researchers 
and  developers.  Objective  6  is,  in  itself,  the  essence  of  the  information  anal¬ 
ysis  center  technology  transfer  process  -  the  ability  to  synthesize  and  distill 
information  about  the  techno-logy. 

The  benefits  of  the  DACS,  when  they  are  realized,  will  result  in  efficient 
and  effective  technology  transfer  (benefits  1-3)  and  a  start  at  escaping  from 
the  software  predicament  which  the  Air  Force,  the  Department  of  Defense,  and 
the  entire  software  community  are  facing  (benefits  4  and  5).  Although  these 
benefits  are  ambitious,  the  framework  of  the  information  analysis  center  pro¬ 
vides  an  excellent  mechanism  for  achieving  these  goals. 

How  DACS  Operates 

Objectives  cannot  be  fulfilled  or  benefits  realized  unless  a  methodology 
to  achieve  these  ends  is  established  and  followed.  Information  analysis  centers 
serve  technology  transfer  by  synthesizing,  distilling,  analyzing  and  repackag¬ 
ing  information.  The  DACS,  as  an  information  analysis  center  technique,  pro¬ 
vides  a  mechanism  for  synthesizing  and  analyzing  the  information  concerning 
software  engineering  technology.  Since  the  DACS  is  currently  a  pilot  facility, 
the  techniques  are  just  being  formatted,  utilized,  and  tested.  Although  these 
techniques  are  still  in  a  test  mode,  they  should  be  adequate  as  they  follow  the 
techniques  used  for  other  technical  IAC's  with  a  fair  amount  of  success.  A 
functional  model  of  the  techniques  is  shown  in  Figure  1. 

DACS  Activities  and  Products 


Two  major  components  make  up  the  technique  represented  in  Figure  1:  (1) 

building  an  information  base  about  software  engineering  technology  and,  (2) 
transferring  information  about  the  technology  in  a  form  that  can  readily  be 
understood,  evaluated,  and  used  to  the  advantage  of  the  software  developer  by 
the  processes  of  information  analysis  and  synthesis.  Each  of  the  major  compon¬ 
ents  consists  of  several  sequential  processing  steps  with  each  processing  step 
resulting  in  the  output  of  a  particular  information  type.  As  a  pilot  facility, 
the  DACS  has  more  experience  with  the  earlier  steps,  but  all  steps  have  been 
utilized  to  some  extent. 

Building  the  technology  information  base  has  been  a  major  effort  and  con¬ 
cern  of  the  DACS  to  date.  This  process  consists  of  (1)  information  collection 
and,  (2)  information  organization.  From  a  technology  transfer  viewpoint,  the 
purpose  of  these  two  steps  is  to  prepare  the  information  base  so  that  it  can 
later  be  synthesized  into  a  form  more  suitable  for  technology  transfer. 
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Information  collection  is  being  actively  pursued  by  the  DACS.  The  profes¬ 
sional  society  publications  are  reviewed,  as  are  conference  proceedings:  re¬ 
ports  on  new  research  projects,  trade  journals,  and  technical  reports  from 
universities,  government,  and  industry.  Information  on  new  and  previous  soft¬ 
ware  engineering  research  is  the  primary  result  of  this  review  and  collection 
procedure.  Software  experience  data  is  also  being  collected  by  the  DACS.  At 
the  present  time,  seven  major  datasets  have  been  assembled  and  two  are  currently 
being  assembled.  Collection  of  all  this  information  is,  of  course,  necessary 
prior  to  any  synthesis  or  analysis  of  the  data  that  is  required  for  technology 
transfer. 

Information  organization  is  the  next  step  in  the  process  of  building  a 
technology  information  base.  Information  management  skills,  such  as  indexing, 
abstracting,  and  information  storage  and  retrieval,  are  used  in  this  phase. 

The  information  collected  in  the  first  step  is  indexed,  abstracted,  and  entered 
into  a  computerized  information  retrieval  system.  Custom  bibliographies  can 
then  be  produced  by  the  retrieval  system  by  specifying  subject  keywords  or  by 
qualifications  on  other  fields  such  as  author  or  title.  These  bibliographies 
are  of  use  to  the  research  community  and  developers  in  locating  results  of  re¬ 
search.  At  this  point  in  the  process,  the  information  is  organized  as  final 
preparation  for  the  steps  of  information  synthesis  and  analysis  that  are  so 
important  to  the  information  analysis  center's  contribution  to  technology  trans¬ 
fer.  These  first  two  steps  (information  collection  and  organization)  are  per¬ 
formed  by  most  technical  libraries.  The  next  two  steps,  however,  separate  the 
information  analysis  center  from  the  technical  library. 

Information  synthesis,  the  next  step,  is  the  core  of  the  technology  trans¬ 
fer  process.  Information  generated  by  the  researcher  must  be  digested  and  made 
palatable  for  the  developer.  If  at  all  possible,  the  information  must  be  pre¬ 
sented  in  a  format  to  allow  cost-benefit  assessment  by  the  developer.  Depend¬ 
ing  on  the  novelty  and  age  of  the  technology,  this  may  not  always  be  possible. 

At  the  very  least,  however,  the  information  must  be  distilled  so  the  basic  con¬ 
cepts  and  principles  are  explained  in  a  language  that  the  developer  can  compre¬ 
hend.  This  is  not  an  easy  task.  Considerable  skills  in  communications  and 
expertise  in  the  technology  are  required  to  produce  the  handbooks  and  state-of- 
the-art  reports  that  are  the  outputs  of  this  technology  transfer  process. 

DACS  has  produced  a  state-of-the-art  report  on  quantitative  software 
models.  >3  Research  in  this  area  has  been  extensive  and  has  the  potential  for 
being  used  by  developers,  so  this  subject  area  was  a  prime  candidate  for  this 
initial  effort.  With  these  thoughts  in  mind,  the  state-of-the-art  report  was 
produced  with  two  major  features  to  facilitate  technology  transfer.  The  first 
feature  was  a  description  of  the  salient  characteristics  of  the  model,  such  as 
data  parameters,  key  equations  and  relationships,  and  experiences  in  using  the 
model.  This  feature  enables  the  developer  to  understand  the  model’s  concepts 
and  capabilities. 

Synthesis  of  information  was  carried  one  step  further  to  produce  the 
second  feature  of  the  report.  A  matrix  was  prepared  to  correlate  model  with 
data  parameters.  By  using  this  matrix,  a  developer  can  quickly  determine  what 
models  could  be  used  with  the  data  parameters  available  to  the  developer.  If 
data  parameters  were  unavailable,  the  cost  of  collecting  them  could  be  weighed 
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against  the  benefits  of  the  model  as  presented  in  the  description. 

Additionally,  DACS  has  published  a  glossary  of  software  engineering 
terms.'4  This  glossary  should  aid  the  technology  transfer  process  by  providing 
a  reference  point  for  uniform  terminology. 

One  last  step  is  included  in  the  technology  transfer  process:  information 
analysis.  Analysis  goes  one  step  beyond  synthesis  by  providing  an  evaluation 
of  the  technology.  As  the  center  makes  the  transfer  from  pilot  to  full-scale 
operation,  this  analysis  effort  will  be  pursued.  One  target  of  analysis  that 
DACS  would  like  to  examine  is  the  effectiveness  of  modern  programming  practices 
At  the  present  time,  data  sources  for  such  an  effort  are  being  collected  and  or¬ 
ganized. 

User  Perceptions 

A  recent  survey  mailed  with  the  DACS  Newsletter  is  currently  being  an¬ 
alyzed.  One  interesting  result  in  the  light  of  the  role  of  the  DACS  as  techno¬ 
logy  transfer  agent,  is  the  interest  of  the  respondents  in  state-of-the-art 
reports.  These  reports,  as  previously  mentioned,  are  one  of  the  primary  pro¬ 
ducts  of  the  DACS. 

Questionnaire  respondents  were  given  a  list  of  the  seven  types  of  informa¬ 
tion  processed  and/or  generated  by  an  information  analysis  center  and  were  ask¬ 
ed  the  following: 

"For  your  job,  please  rate  the  value  of  the  following  types  of 

information.  (1  =  most  valuable)" 

Table  I  summarizes  the  results.  Although  these  results  are  preliminary, 
the  results  point  to  a  definite  preference  for  surveys  of  current  technologies. 
The  final  results  of  che  survey  will  be  used  to  plan  future  DACS  services  and 
products. 

Conclusion 


The  need  for  technology  transfer  in  software  engineering  is  clear  and 
essential  if  the  software  world  is  to  escape  from  its  present  predicament. 
Various  methodologies  and  mechanisms  exist  for  this  transfer  process.  One 
process,  using  an  information  analysis  center  as  a  transfer  agent,  is  appeal¬ 
ing  because  the  capability  of  the  information  analysis  center  as  synthesizer 
of  technology  fits  the  task  of  transferring  technology  in  such  a  way  that  the 
software  developer  and  user  can  understand  and  evaluate  the  technology. 

As  a  software  engineering  information  analysis  center,  the  DACS  is  fulfilling 
the  role  of  transfer  agent.  Mechanisms  are  in  place  to  synthesize  the  software 
engineering  technology  and  disseminate  it  to  software  developers.  During  its 
operation,  DACS  has  built  an  information  base  of  software  engineering  and  used 
that  base  to  synthesize  the  technology  into  several  handbooks  and  state-of-the- 
art  reports.  Response  to  the  reports  have  been  encouraging,  and  the  DACS  plans 
to  issue  more  in  its  role  as  a  software  engineering  technology  transfer  agent. 
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TABLE  I  USER  SURVEY  -  PRELIMINARY  RESULTS 


Profile  of  user: 

User 

Percent  of  Respondees 

Researchers: 

28% 

Developers: 

70% 

(Managers  38%,  Programmer/analyst  32% 

Not  designated: 

3% 

100% 

Information  value: 

Percent  of  respondees  ranking  an  information  type  as  1,  2,  or  3 
on  a  scale  of  1  (most  valuable)  to  7  (least  valuable). 


Information  Type  Percent 

Information  on  new  software  research  57% 

State-of-the-art  reports  75% 

Bibliographies  15% 

Software  experience  data  24% 

Information  on  previous  software 

research  35% 

Handbook  information  37% 

Critical  reviews  37% 
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5  MISSION  j 

j  Rome  Air  Development  Center  2 

S  WtW  p£an6  and  executes  research,  development,  test  and  v 

*  selected  acquisition  pA.ogA.ams  in  support  oh  Command,  Control  v 

2  Cornmn-ccattona  and  Intelligence  |C3I)  activities .  Technical  ? 
%  eng-tnee*-tng  4uppoa*  within  areas  oh  technical  competence  C 
N  -cA  pxovided  to  ESP  Paogaam  0^-cc.eA  (  POa  )  and  o-Cfeea  ESP  $ 

J  elements.  The  principal  technical  mission  areas  axe  x 

S  communications ,  electromagnetic  guidance  and  control,  Sux-  s> 
5  ve^i«ce  oh  ground  and  aerospace  objects,  intelligence  data  J 

•  collection  and  handling,  information  system  technology,  w 

k  ionosphere c  propagation,  solid  state  sciences,  microwave  ^ 
b  physics  and  electronic  reliability,  maintainability  and 
S  compatibility. 
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